Add files using upload-large-folder tool

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/README.md

@@ -0,0 +1,70 @@
+# mmlu_pro_plus
+
+### Paper
+
+Title: `MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs`
+
+Abstract: `Existing benchmarks for large language models (LLMs) increasingly struggle to differentiate between
+top-performing models, underscoring the need for more challenging evaluation frameworks.
+We introduce MMLU-Pro+, an enhanced benchmark building upon MMLU-Pro to assess shortcut
+learning and higher-order reasoning in LLMs. By incorporating questions with multiple
+correct answers across diverse domains, MMLU-Pro+ tests LLMs' ability to engage in complex
+reasoning and resist simplistic problem-solving strategies. Our results show that
+MMLU-Pro+ maintains MMLU-Pro's difficulty while providing a more rigorous test of
+model discrimination, particularly in multi-correct answer scenarios.
+We introduce novel metrics like shortcut selection ratio and correct pair identification
+ratio, offering deeper insights into model behavior and anchoring bias.
+Evaluations of six state-of-the-art LLMs reveal significant performance gaps,
+highlighting variations in reasoning abilities and bias susceptibility.`
+
+Homepage: https://github.com/asgsaeid/mmlu-pro-plus
+
+### Citation
+
+```bibtex
+@article{taghanaki2024mmlu,
+  title={MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs},
+  author={Taghanaki, Saeid Asgari and Khani, Aliasgahr and Khasahmadi, Amir},
+  journal={arXiv preprint arXiv:2409.02257},
+  year={2024}
+}
+```
+
+### Groups and Tasks
+
+#### Groups
+
+* `mmlu_pro_plus`: 'All 14 subjects of the mmlu_pro_plus dataset, evaluated following the methodology in mmlu's original implementation'
+
+#### Tasks
+
+The following tasks evaluate subjects in the mmlu_pro dataset
+- `mmlu_pro_plus_biology`
+- `mmlu_pro_plus_business`
+- `mmlu_pro_plus_chemistry`
+- `mmlu_pro_plus_computer_science`
+- `mmlu_pro_plus_economics`
+- `mmlu_pro_plus_engineering`
+- `mmlu_pro_plus_health`
+- `mmlu_pro_plus_history`
+- `mmlu_pro_plus_law`
+- `mmlu_pro_plus_math`
+- `mmlu_pro_plus_other`
+- `mmlu_pro_plus_philosophy`
+- `mmlu_pro_plus_physics`
+- `mmlu_pro_plus_psychology`
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+* [x] Is the task an existing benchmark in the literature?
+  * [x] Have you referenced the original paper that introduced the task?
+  * [x] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+
+If other tasks on this dataset are already supported:
+* [ ] Is the "Main" variant of this task clearly denoted?
+* [x] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [x] Have you noted which, if any, published evaluation setups are matched by this variant?
+
+### Changelog

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/_default_template_yaml

@@ -0,0 +1,33 @@
+dataset_path: saeidasgari/mmlu-pro-plus
+test_split: test
+fewshot_split: validation
+fewshot_config:
+  sampler: first_n
+  doc_to_text: !function utils.fewshot_to_text
+  doc_to_target: ""
+output_type: generate_until
+doc_to_text: !function utils.doc_to_text
+doc_to_target: answer
+filter_list:
+  - name: "custom-extract"
+    filter:
+      - function: "regex"
+        regex_pattern: 'answer is \(?([ABCDEFGHIJKL])\)?'
+        # regex_pattern: r".*[aA]nswer:\s*([A-L])",
+      - function: "take_first"
+generation_kwargs:
+  until:
+    - "</s>"
+    - "Q:"
+    - "<|im_end|>"
+  do_sample: false
+  temperature: 0.0
+num_fewshot: 5
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+    ignore_case: true
+    ignore_punctuation: true
+metadata:
+  version: 1.0

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/_mmlu_pro_plus.yaml

@@ -0,0 +1,23 @@
+group: mmlu_pro_plus
+task:
+  - mmlu_pro_plus_biology
+  - mmlu_pro_plus_business
+  - mmlu_pro_plus_chemistry
+  - mmlu_pro_plus_computer_science
+  - mmlu_pro_plus_economics
+  - mmlu_pro_plus_engineering
+  - mmlu_pro_plus_health
+  - mmlu_pro_plus_history
+  - mmlu_pro_plus_law
+  - mmlu_pro_plus_math
+  - mmlu_pro_plus_other
+  - mmlu_pro_plus_philosophy
+  - mmlu_pro_plus_physics
+  - mmlu_pro_plus_psychology
+aggregate_metric_list:
+  - aggregation: mean
+    metric: exact_match
+    weight_by_size: true
+    filter_list: custom-extract
+metadata:
+  version: 1.0

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_biology.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about biology. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_biology"
+task_alias: "biology"
+process_docs: !function utils.process_biology

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_business.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about business. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_business"
+task_alias: "business"
+process_docs: !function utils.process_business

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_chemistry.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about chemistry. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_chemistry"
+task_alias: "chemistry"
+process_docs: !function utils.process_chemistry

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_computer_science.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about computer science. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_computer_science"
+task_alias: "computer_science"
+process_docs: !function utils.process_computer_science

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_economics.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about economics. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_economics"
+task_alias: "economics"
+process_docs: !function utils.process_economics

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_engineering.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about engineering. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_engineering"
+task_alias: "engineering"
+process_docs: !function utils.process_engineering

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_health.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about health. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_health"
+task_alias: "health"
+process_docs: !function utils.process_health

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_history.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about history. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_history"
+task_alias: "history"
+process_docs: !function utils.process_history

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_law.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about law. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_law"
+task_alias: "law"
+process_docs: !function utils.process_law

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_math.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about math. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_math"
+task_alias: "math"
+process_docs: !function utils.process_math

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_other.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about other. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_other"
+task_alias: "other"
+process_docs: !function utils.process_other

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_philosophy.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about philosophy. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_philosophy"
+task_alias: "philosophy"
+process_docs: !function utils.process_philosophy

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_physics.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about physics. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_physics"
+task_alias: "physics"
+process_docs: !function utils.process_physics

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/mmlu_pro_plus_psychology.yaml

@@ -0,0 +1,5 @@
+description: "The following are multiple choice questions (with answers) about psychology. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n"
+include: "_default_template_yaml"
+task: "mmlu_pro_plus_psychology"
+task_alias: "psychology"
+process_docs: !function utils.process_psychology

lm-evaluation-harness/lm_eval/tasks/mmlu-pro-plus/utils.py

@@ -0,0 +1,63 @@
+from functools import partial
+
+
+choices = [
+    "A",
+    "B",
+    "C",
+    "D",
+    "E",
+    "F",
+    "G",
+    "H",
+    "I",
+    "J",
+    "K",
+    "L",
+    "M",
+    "N",
+    "O",
+    "P",
+]
+
+
+def format_cot_example(example, including_answer=True):
+    prompt = "Question:\n"
+    question = example["question"]
+    options = example["options"]
+    prompt += question + "\n"
+    prompt += "Options:\n"
+    for i, opt in enumerate(options):
+        prompt += "{}. {}\n".format(choices[i], opt)
+    if including_answer:
+        cot_content = example["cot_content"].replace(
+            "A: Let's think step by step.", "Answer: Let's think step by step."
+        )
+        prompt += cot_content + "\n\n"
+    else:
+        prompt += "Answer: Let's think step by step."
+    return prompt
+
+
+doc_to_text = partial(format_cot_example, including_answer=False)
+fewshot_to_text = partial(format_cot_example, including_answer=True)
+
+
+def process_docs(dataset, subject):
+    return dataset.filter(lambda x: x["category"] == subject)
+
+
+process_biology = partial(process_docs, subject="biology")
+process_business = partial(process_docs, subject="business")
+process_chemistry = partial(process_docs, subject="chemistry")
+process_computer_science = partial(process_docs, subject="computer science")
+process_economics = partial(process_docs, subject="economics")
+process_engineering = partial(process_docs, subject="engineering")
+process_health = partial(process_docs, subject="health")
+process_history = partial(process_docs, subject="history")
+process_law = partial(process_docs, subject="law")
+process_math = partial(process_docs, subject="math")
+process_other = partial(process_docs, subject="other")
+process_philosophy = partial(process_docs, subject="philosophy")
+process_physics = partial(process_docs, subject="physics")
+process_psychology = partial(process_docs, subject="psychology")

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_high_school_us_history.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "high_school_us_history"
+"description": "The following are multiple choice questions (with answers) about high\
+  \ school us history.\n\n"
+"tag": "mmlu_humanities_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_high_school_us_history"
+"task_alias": "high_school_us_history"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_management.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "management"
+"description": "The following are multiple choice questions (with answers) about management.\n\
+  \n"
+"tag": "mmlu_other_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_management"
+"task_alias": "management"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_marketing.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "marketing"
+"description": "The following are multiple choice questions (with answers) about marketing.\n\
+  \n"
+"tag": "mmlu_other_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_marketing"
+"task_alias": "marketing"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_moral_disputes.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "moral_disputes"
+"description": "The following are multiple choice questions (with answers) about moral\
+  \ disputes.\n\n"
+"tag": "mmlu_humanities_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_moral_disputes"
+"task_alias": "moral_disputes"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_moral_scenarios.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "moral_scenarios"
+"description": "The following are multiple choice questions (with answers) about moral\
+  \ scenarios.\n\n"
+"tag": "mmlu_humanities_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_moral_scenarios"
+"task_alias": "moral_scenarios"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_nutrition.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "nutrition"
+"description": "The following are multiple choice questions (with answers) about nutrition.\n\
+  \n"
+"tag": "mmlu_other_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_nutrition"
+"task_alias": "nutrition"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_professional_psychology.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "professional_psychology"
+"description": "The following are multiple choice questions (with answers) about professional\
+  \ psychology.\n\n"
+"tag": "mmlu_social_sciences_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_professional_psychology"
+"task_alias": "professional_psychology"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_public_relations.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "public_relations"
+"description": "The following are multiple choice questions (with answers) about public\
+  \ relations.\n\n"
+"tag": "mmlu_social_sciences_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_public_relations"
+"task_alias": "public_relations"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_security_studies.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "security_studies"
+"description": "The following are multiple choice questions (with answers) about security\
+  \ studies.\n\n"
+"tag": "mmlu_social_sciences_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_security_studies"
+"task_alias": "security_studies"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_sociology.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "sociology"
+"description": "The following are multiple choice questions (with answers) about sociology.\n\
+  \n"
+"tag": "mmlu_social_sciences_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_sociology"
+"task_alias": "sociology"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_us_foreign_policy.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "us_foreign_policy"
+"description": "The following are multiple choice questions (with answers) about us\
+  \ foreign policy.\n\n"
+"tag": "mmlu_social_sciences_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_us_foreign_policy"
+"task_alias": "us_foreign_policy"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_virology.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "virology"
+"description": "The following are multiple choice questions (with answers) about virology.\n\
+  \n"
+"tag": "mmlu_other_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_virology"
+"task_alias": "virology"

lm-evaluation-harness/lm_eval/tasks/mmlu/default/mmlu_world_religions.yaml

@@ -0,0 +1,7 @@
+"dataset_name": "world_religions"
+"description": "The following are multiple choice questions (with answers) about world\
+  \ religions.\n\n"
+"tag": "mmlu_humanities_tasks"
+"include": "_default_template_yaml"
+"task": "mmlu_world_religions"
+"task_alias": "world_religions"

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_college_physics.yaml

@@ -0,0 +1,61 @@
+dataset_name: college_physics
+description: The following are multiple choice questions (with answers) about college
+  physics.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'A refracting telescope consists of two converging lenses separated by
+      100 cm. The eye-piece lens has a focal length of 20 cm. The angular magnification
+      of the telescope is
+
+      (A) 4 (B) 5 (C) 6 (D) 20'
+    target: Let's think step by step. In a refracting telescope, if both lenses are
+      converging, the focus of both lenses must be between the two lenses, and thus
+      the focal lengths of the two lenses must add up to their separation. Since the
+      focal length of one lens is 20 cm, the focal length of the other must be 80
+      cm. The magnification is the ratio of these two focal lengths, or 4. The answer
+      is (A).
+  - question: 'The muon decays with a characteristic lifetime of about 10^-6 second into
+      an electron, a muon neutrino, and an electron antineutrino. The muon is forbidden
+      from decaying into an electron and just a single neutrino by the law of conservation
+      of
+
+      (A) charge (B) mass (C) energy and momentum (D) lepton number'
+    target: Let's think step by step. Lepton number must be conserved, meaning the
+      total number of leptons minus the number of antileptons. If a muon decays into
+      an electron and a single neutrino, the total lepton number would go from one
+      to two, violating lepton number conservation. The answer is (D).
+  - question: 'One end of a Nichrome wire of length 2L and cross-sectional area A is
+      attached to an end of another Nichrome wire of length L and cross- sectional
+      area 2A. If the free end of the longer wire is at an electric potential of 8.0
+      volts, and the free end of the shorter wire is at an electric potential of 1.0
+      volt, the potential at the junction of the two wires is most nearly equal to
+
+      (A) 2.4 V (B) 3.3 V (C) 4.5 V (D) 5.7 V'
+    target: Let's think step by step. This is a simple voltage divider problem, where
+      the longer wire has a resistance four times that of the shorter end. So the
+      voltage divider ratio is 1 / 5, meaning that the potential in the middle is
+      1.0 V + (8.0 V - 1.0 V) * 1/5 = 2.4 V. The answer is (A).
+  - question: 'A refracting telescope consists of two converging lenses separated by
+      100 cm. The eye-piece lens has a focal length of 20 cm. The angular magnification
+      of the telescope is
+
+      (A) 4 (B) 5 (C) 6 (D) 20'
+    target: Let's think step by step. In a refracting telescope, if both lenses are
+      converging, the focus of both lenses must be between the two lenses, and thus
+      the focal lengths of the two lenses must add up to their separation. Since the
+      focal length of one lens is 20 cm, the focal length of the other must be 80
+      cm. The magnification is the ratio of these two focal lengths, or 4. The answer
+      is (A).
+  - question: 'For which of the following thermodynamic processes is the increase in
+      the internal energy of an ideal gas equal to the heat added to the gas?
+
+      (A) Constant temperature (B) Constant volume (C) Constant pressure (D) Adiabatic'
+    target: 'Let''s think step by step. Heat added to the gas can go into the gases
+      internal energy or work done against an external force. However, if the volume
+      of the gas container is constant, no work will be done (since work is pressure
+      times change in volume). So, at constant volume, all of the heat goes into the
+      internal energy. The answer is (B).'
+tag: mmlu_flan_cot_fewshot_stem
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_college_physics

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_computer_security.yaml

@@ -0,0 +1,50 @@
+dataset_name: computer_security
+description: The following are multiple choice questions (with answers) about computer
+  security.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'SHA-1 has a message digest of
+
+      (A) 160 bits (B) 512 bits (C) 628 bits (D) 820 bits'
+    target: Let's think step by step. Since SHA-1 is a hash function which takes an
+      question and produces a 160-bit (20-byte) hash value, its message digest is 160
+      bits. The answer is (A).
+  - question: "_____________ can modify data on your system \u2013 so that your system\
+      \ doesn\u2019t run correctly or you can no longer access specific data, or it\
+      \ may even ask for ransom in order to give your access.\n(A) IM \u2013 Trojans\
+      \ (B) Backdoor Trojans (C) Trojan-Downloader (D) Ransom Trojan"
+    target: Let's think step by step. The system is asking for trojans, which are
+      for ransom, which means ransom trojan. The answer is (D).
+  - question: 'What is ethical hacking?
+
+      (A) "Hacking" ethics so they justify unintended selfish behavior (B) Hacking
+      systems (e.g., during penetration testing) to expose vulnerabilities so they
+      can be fixed, rather than exploited (C) Hacking into systems run by those whose
+      ethics you disagree with (D) A slang term for rapid software development, e.g.,
+      as part of hackathons'
+    target: Let's think step by step. Ethical hacking is a process of detecting vulnerabilities
+      in an application, system, or organization's infrastructure that an attacker
+      can use to exploit an individual or organization. They use this process to prevent
+      cyberattacks and security breaches by lawfully hacking into the systems and
+      looking for weak points. The answer is (B).
+  - question: 'The ____________ is anything which your search engine cannot search.
+
+      (A) Haunted web (B) World Wide Web (C) Surface web (D) Deep Web'
+    target: "Let's think step by step. The search engine searches on the Surface Web,\
+      \ which is the portion of the world wide web which is visible so (B,C) are wrong.\
+      \ The Haunted Web doesn\u2019t correspond to an internet concept. The Deep Web\
+      \ is the part of the World Wide Web which is not indexed. The answer is (D)."
+  - question: 'Exploitation of the Heartbleed bug permits
+
+      (A) overwriting cryptographic keys in memory (B) a kind of code injection (C)
+      a read outside bounds of a buffer (D) a format string attack'
+    target: 'Let''s think step by step. The Heartbleed Bug is a serious vulnerability
+      in the popular OpenSSL cryptographic software library. Heartbleed resulted from
+      improper question validation (due to a missing bounds check) in the implementation
+      of the TLS heartbeat extension. The vulnerability was classified as a buffer
+      over-read, a situation where more data can be read than should be allowed. The
+      answer is (C).'
+tag: mmlu_flan_cot_fewshot_stem
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_computer_security

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_conceptual_physics.yaml

@@ -0,0 +1,49 @@
+dataset_name: conceptual_physics
+description: '
+
+  The following are multiple choice questions (with answers) about conceptual physics.'
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'Colors in a soap bubble result from light
+
+      (A) converted to a different frequency (B) deflection (C) interference (D) polarization'
+    target: Let's think step by step. In a soap bubble film, the light bounces between
+      the two soap-air interfaces many times, interfering with itself constructively
+      or destructively depending on the width of the film. This results in different
+      colors being visible. The answer is (C).
+  - question: 'Compared with the mass of a uranium atom undergoing fission, the combined
+      masses of the products after fission are
+
+      (A) less (B) more (C) the same (D) zero'
+    target: Let's think step by step. Fission releases energy, which comes from the
+      rest mass of its initial nucleus. Thus the mass of the products is less than
+      the mass of the reactant uranium nucleus. The answer is (A).
+  - question: 'Things that are equivalent according to the equivalence principle are
+
+      (A) space and time. (B) a traveling twin and a stay-at-home twin. (C) gravity
+      and acceleration. (D) mass and energy.'
+    target: "Let's think step by step. Einstein\u2019s famous equivalence principle\
+      \ states that gravity and acceleration are equivalent. The answer is (C)."
+  - question: 'Which of these three elements has the most mass per nucleon?
+
+      (A) Hydrogen (B) Iron (C) Uranium (D) Same in each'
+    target: Let's think step by step. Due to nuclear binding energy, the mass of an
+      atomic nucleus is less than the sum of individual masses of the free constituent
+      protons and neutrons; this is known as the mass defect. Hydrogen has no mass
+      defect because it has only a single nucleon, so it will have the most mass per
+      nucleon. The answer is (A).
+  - question: 'A model airplane flies slower when flying into the wind and faster with
+      wind at its back. When launched at right angles to the wind a cross wind its
+      groundspeed compared with flying in still air is
+
+      (A) the same (B) greater (C) less (D) either greater or less depending on wind
+      speed'
+    target: "Let's think step by step. The plane\u2019s speed in the direction of\
+      \ the wind is greater than it would be in the absence of wind, and its direction\
+      \ orthogonal to the wind is the same as it would be in the absence of the wind.\
+      \ The total speed, which is these two components added in quadrature, is thus\
+      \ greater than the speed in still air. The answer is (B).\n\n"
+tag: mmlu_flan_cot_fewshot_stem
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_conceptual_physics

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_elementary_mathematics.yaml

@@ -0,0 +1,77 @@
+dataset_name: elementary_mathematics
+description: The following are multiple choice questions (with answers) about elementary
+  mathematics.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'Olivia used the rule "Add 11" to create the number pattern shown below.
+      10, 21, 32, 43, 54. Which statement about the number pattern is true?
+
+      (A) The 10th number in the pattern will be an even number.
+
+      (B) The number pattern will never have two even numbers next to each other.
+
+      (C) The next two numbers in the pattern will be an even number then an odd number.
+
+      (D) If the number pattern started with an odd number then the pattern would
+      have only odd numbers in it.'
+    target: Let's think step by step. Choice A is incorrect because every even-numbered
+      term in the pattern is odd, and 10 is an even number. Choice B is correct, because
+      adding an odd number (in this case 11) to an odd number produces an even number,
+      and adding an odd number to an even number produces an odd number. Thus the
+      terms in the pattern will alternate between odd and even, so there will never
+      be two even numbers next to each other. Choice C is incorrect because the last
+      term in the example is even (54), and we know that the terms will alternate
+      between even and odd. Choice D is incorrect because the terms in the pattern
+      will alternate between odd and even, regardless of the value of the first term.
+      The answer is (B).
+  - question: 'The population of the city where Michelle was born is 145,826. What is
+      the value of the 5 in the number 145,826?
+
+      (A) 5 thousands
+
+      (B) 5 hundreds
+
+      (C) 5 tens
+
+      (D) 5 ones'
+    target: 'Let''s think step by step. Choice A is correct, because there are three
+      digits following the 5, so
+
+      the 5 is in the thousands place. Thus the other choices are incorrect. The answer
+      is (A).'
+  - question: "A store sells 107 different colors of paint. They have 25 cans of each\
+      \ color in storage. The number of cans of paint the store has in storage can\
+      \ be found using the expression below. 107 \xD7 25. How many cans of paint does\
+      \ the store have in storage?\n(A) 749\n(B) 2,675\n(C) 2,945\n(D) 4,250"
+    target: Let's think step by step. We can calculate 107 x 25 = (100 x 25) + (7
+      x 25) = 2500 + 175 = 2675. The answer is (B).
+  - question: 'A total of 30 players will play basketball at a park. There will be exactly
+      5 players on each team. Which statement correctly explains how to find the number
+      of teams needed?
+
+      (A) Add 5 to 30 to find 35 teams.
+
+      (B) Divide 30 by 5 to find 6 teams.
+
+      (C) Multiply 30 and 5 to find 150 teams.
+
+      (D) Subtract 5 from 30 to find 25 teams.'
+    target: Let's think step by step. We want to find the number of teams. We know
+      that there are 5 players/team, and 30 players. Thus to get the number of teams
+      we divide players by players/team, so 30 players / 5 players/team = 6 teams.
+      The answer is (B).
+  - question: 'Which expression is equivalent to 5 x 9?
+
+      (A) (5 x 4) x (6 x 5)
+
+      (B) (5 x 5) + (5 x 4)
+
+      (C) (5 x 5) + (5 x 9)
+
+      (D) (5 x 9) x (6 x 9)'
+    target: 'Let''s think step by step. We know that 9 = (5 + 4), so 5 x 9 = 5 x (5
+      + 4) = (5 x 5) + (5 x 4). The answer is (B).'
+tag: mmlu_flan_cot_fewshot_stem
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_elementary_mathematics

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_formal_logic.yaml

@@ -0,0 +1,70 @@
+dataset_name: formal_logic
+description: The following are multiple choice questions (with answers) about formal
+  logic.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: "Which of the given formulas of PL is the best symbolization of the following\
+      \ sentence?\nTurtles live long lives and are happy creatures, unless they are\
+      \ injured.\n(A) (L \u2022 H) \u2261 I (B) (L \u2022 H) \u2228 I (C) L \u2022\
+      \ (H \u2228 I) (D) L \u2022 (H \u2283 R)."
+    target: "Let's think step by step. We refer to Wikipedia articles on formal logic\
+      \ for help. Let\u2019s solve this step by step. Let \u201CL\u201D denote \u201C\
+      living long\u201D, H \u201Cbeing happy\u201D, and \u201CI\u201D \u201Cbeing\
+      \ injured\u201D. Now, consider each choice:\n(A) means (living long AND being\
+      \ happy) is equivalent to (being injured). \n(B) means (living long AND being\
+      \ happy) OR (being injured). \n(C) means (living long) AND (being happy OR being\
+      \ injured). \n(D) means (living long) AND (being happy implies being R), but\
+      \ what R denotes is not clear.\nObviously, (B) is the best symbolization of\
+      \ the original sentence. The answer is (B)."
+  - question: 'Select the best translation into predicate logic.George borrows Hector''s
+      lawnmower. (g: George; h: Hector; l: Hector''s lawnmower; Bxyx: x borrows y
+      from z).
+
+      (A) Blgh (B) Bhlg (C) Bglh (D) Bghl'
+    target: "Let's think step by step. We refer to Wikipedia articles on formal logic\
+      \ for help. Let\u2019s solve this step by step. We are told that \u201CBxyx\u201D\
+      \ means \u201Cx borrows y from z\u201D. We can rewrite \u201CGeorge borrows\
+      \ Hector's lawnmower\u201D as \u201CGeorge borrows a lawnmower from Hector\u201D\
+      , which can then be translated into predicate logic as \u201CBglh\u201D. The\
+      \ answer \u201CBglh\u201D appears in (C); therefore, (C) must be the correct\
+      \ answer. The answer is (C)."
+  - question: "\nSelect the best English interpretation of the given arguments in predicate\
+      \ logic.\nDm\n(\u2200x)(Wx \u2283 ~Dx). \n(\u2200x)Wx \u2228 Ag\t/ (\u2203x)Ax\n\
+      (A) Marina is a dancer. Some weaklings are not dancers. Either everything is\
+      \ a weakling or Georgia plays volleyball. So something plays volleyball. (B)\
+      \ Marina is a dancer. No weakling is a dancer. Everything is either a weakling\
+      \ or plays volleyball. So something plays volleyball. (C) Marina is a dancer.\
+      \ Some weaklings are not dancers. Everything is either a weakling or plays volleyball.\
+      \ So something plays volleyball. (D) Marina is a dancer. No weakling is a dancer.\
+      \ Either everything is a weakling or Georgia plays volleyball. So something\
+      \ plays volleyball."
+    target: "Let's think step by step. We refer to Wikipedia articles on formal logic\
+      \ for help. Let\u2019s solve this step by step. Let \u201CD\u201D denote \u201C\
+      being a dancer\u201D, \u201Cm\u201D denote \u201CMaria\u201D, \u201Cg\u201D\
+      \ denote \u201CGeorgia\u201D, \u201CW\u201D denote \u201Cweakling\u201D, \u201C\
+      A\u201D denote \u201Cplaying volleyball\u201D. Then, we have the following:\n\
+      1. Dm \u2192 Maria is a dance.\n2. (\u2200x)(Wx \u2283 ~Dx). \u2192 For all\
+      \ x, if x is a weakling, then x is not a dancer. In other words, no weakling\
+      \ is a dancer.\n3. (\u2200x)Wx \u2228 Ag\t/ (\u2203x)Ax \u2192 For all x, x\
+      \ is a weakling or Georgia plays volleyball. So there exists an x that plays\
+      \ volleyball. \nOptions (A) and (C) do claim that some weaklings are not dancers,\
+      \ but the second argument strongly states that no weakling is a dancer. Thus,\
+      \ we can eliminate them. Option (B) omits the important detail about Georgia\
+      \ playing volleyball. Option (D) has all the details presented in the arguments\
+      \ and is the best English interpretation of the arguments. The answer is (D)."
+  - question: "Select the best translation into predicate logic: No people drive on Mars.\n\
+      (A) ~Pd (B) (\u2200x)(Px \u2228 ~Dx) (C) (\u2200x)(Px \u2283 ~Dx) (D) ~Dp"
+    target: "Let's think step by step. We refer to Wikipedia articles on formal logic\
+      \ for help. Let\u2019s solve this step by step. Let \u201CP\u201D denote \u201C\
+      being on Mars\u201D and \u201CD\u201D denote \u201Cdriving on Mars\u201D. Then\
+      \ let\u2019s consider each option:\nOption (A): ~Pd \u2192 d is not on Mars.\n\
+      Option (B): (\u2200x)(Px \u2228 ~Dx) \u2192 For all x, x is on Mars and x do\
+      \ not drive on Mars.\nOption (C): (\u2200x)(Px \u2283 ~Dx) \u2192 For all x,\
+      \ x is on Mars implies that x do not drive on Mars.\nOption (D): ~Dp: \u2192\
+      \ p do not drive on Mars.\nOf all these options, Option (C) appears to be the\
+      \ best and most meaningful interpretation of the argument \u201CNo people drive\
+      \ on Mars.\u201D The answer is (C).\n\n"
+tag: mmlu_flan_cot_fewshot_humanities
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_formal_logic

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_global_facts.yaml

@@ -0,0 +1,49 @@
+dataset_name: global_facts
+description: The following are multiple choice questions (with answers) about global
+  facts.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: "As of 2017, how many of the world\u2019s 1-year-old children today have\
+      \ been vaccinated against some disease? *\n(A) 80% (B) 60% (C) 40% (D) 20%"
+    target: Let's think step by step. We refer to Wikipedia articles on global facts
+      for help. According to data published by the World Health Organization, the
+      nummber of 1-year-old children vaccinated in 2017 exceeds 80%. The answer is
+      (A).
+  - question: 'As of 2019, about what percentage of Americans agree that the state is
+      run for the benefit of all the people?
+
+      (A) 31% (B) 46% (C) 61% (D) 76%'
+    target: Let's think step by step. We refer to Wikipedia articles on global facts
+      for help. In 2019, about 46% percentage of Americans agree that the state is
+      run for the benefit of all the people. The answer is (B).
+  - question: 'As of 2019, about what percentage of Russians say it is very important
+      to have free media in our country without government/state censorship?
+
+      (A) 38% (B) 53% (C) 68% (D) 83%'
+    target: Let's think step by step. We refer to Wikipedia articles on global facts
+      for help. As of 2019, about 38% of Russians say it is very important to have
+      free media in our country. The answer is (A).
+  - question: 'As of 2015, since 1990 forests have ____ in Europe and have ____ in Africa
+      and the Americas.
+
+      (A) increased, increased (B) increased, decreased (C) decreased, increased (D)
+      decreased, decreased'
+    target: Let's think step by step. We refer to Wikipedia articles on global facts
+      for help. As of 2015, since 1990 forests have increased in Europe and have decreased
+      in Africa and the Americas. The answer is (B).
+  - question: 'Which of the following pairs of statements are both true (as of 2019)?
+
+      (A) People tend to be optimistic about their own future and the future of their
+      nation or the world. (B) People tend to be optimistic about their own future
+      but pessimistic about the future of their nation or the world. (C) People tend
+      to be pessimistic about their own future but optimistic about the future of
+      their nation or the world. (D) People tend to be pessimistic about their own
+      future and the future of their nation or the world.'
+    target: 'Let''s think step by step. We refer to Wikipedia articles on global facts
+      for help. As of 2019, most people tend to be optimistic about their own future
+      but pessimistic about the future of their nation or the world. The answer is
+      (B).'
+tag: mmlu_flan_cot_fewshot_other
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_global_facts

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_biology.yaml

@@ -0,0 +1,69 @@
+dataset_name: high_school_biology
+description: The following are multiple choice questions (with answers) about high
+  school biology.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: "In animal cells, which of the following represents the most likely pathway\
+      \ that a secretory protein takes as it is synthesized in a cell?\n(A) Plasma\
+      \ membrane\u2013Golgi apparatus\u2013ribosome\u2013secretory vesicle\u2013rough\
+      \ ER (B) Ribosome\u2013Golgi apparatus\u2013rough ER\u2013secretory vesicle\u2013\
+      plasma membrane (C) Plasma membrane\u2013Golgi apparatus\u2013ribosome\u2013\
+      secretory vesicle\u2013rough ER (D) Ribosome\u2013rough ER\u2013Golgi apparatus\u2013\
+      secretory vesicle\u2013plasma membrane"
+    target: Let's think step by step. Protein synthesis starts at the ribosome, so
+      we can eliminate (A) and (C). The ribosome is often in the endoplasmic reticulum
+      and moves from there to the Golgi apparatus, where it is modified and packaged
+      into a vesicle. The vesicle then floats to the plasma membrane and is secreted.
+      The answer is (D).
+  - question: "A mutation in a bacterial enzyme changed a previously polar amino acid\
+      \ into a nonpolar amino acid. This amino acid was located at a site distant\
+      \ from the enzyme\u2019s active site. How might this mutation alter the enzyme\u2019\
+      s substrate specificity?\n(A) By changing the enzyme\u2019s pH optimum (B) By\
+      \ changing the enzyme\u2019s location in the cell (C) By changing the shape\
+      \ of the protein (D) An amino acid change away from the active site cannot alter\
+      \ the enzyme\u2019s substrate specificity."
+    target: Let's think step by step. A change in an amino acid leads to a change
+      in the primary structure of the protein. A change in the primary structure may
+      lead to a change in the secondary and the tertiary structure of the protein.
+      A change in the tertiary structure means a change in the shape of the protein,
+      so (C) has to be correct. Since the change does not affect the active site of
+      the enzyme, we do not expect the activity of the enzyme to be affected. The
+      answer is (C).
+  - question: 'Which of the following is not a way to form recombinant DNA?
+
+      (A) Translation (B) Conjugation (C) Specialized transduction (D) Transformation'
+    target: 'Let''s think step by step. The introduction of foreign DNA or RNA into
+      bacteria or eukaryotic cells is a common technique in molecular biology and
+      scientific research. There are multiple ways foreign DNA can be introduced into
+      cells including transformation, transduction, conjugation, and transfection.
+      In contrast, (A) is not a way to form DNA: during translation the ribosomes
+      synthesize proteins from RNA. The answer is (A).'
+  - question: 'Homologous structures are often cited as evidence for the process of natural
+      selection. All of the following are examples of homologous structures EXCEPT
+
+      (A) the wings of a bird and the wings of a bat (B) the flippers of a whale and
+      the arms of a man (C) the pectoral fins of a porpoise and the flippers of a
+      seal (D) the forelegs of an insect and the forelimbs of a dog'
+    target: "Let's think step by step. \u200B\u200BHomologous structures are similar\
+      \ physical features in organisms that share a common ancestor \u200B\u200Bbut\
+      \ different functions. Comparisons (B) and (C) are clearly homologous because\
+      \ they share a common ancestor and the structures serve different purposes.\
+      \ Bat wings and birg wings are also homologous, while they are both wings, the\
+      \ forelimbs serve different purposes. Insects and dogs are very far ancestors\
+      \ since one is vertebrate while the other is invertebrate and the forelimbs\
+      \ serve the same purpose, so they are not homologous. The answer is (D)."
+  - question: 'Which of the following is not known to be involved in the control of cell
+      division?
+
+      (A) Cyclins (B) Protein kinases (C) Checkpoints (D) Fibroblast cells'
+    target: 'Let''s think step by step. Normal cells move through the cell cycle in
+      a regulated way. At the checkpoint stage, they use information about their own
+      internal state and cues from the environment around them to decide whether to
+      proceed with cell division. Cues like these act by changing the activity of
+      core cell cycle regulators inside the cell. The most common regulators are cyclins
+      and cyclin-dependent kinases. Fibroblast cells do not play any role in cell
+      division. The answer is (D).'
+tag: mmlu_flan_cot_fewshot_stem
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_biology

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_computer_science.yaml

@@ -0,0 +1,84 @@
+dataset_name: high_school_computer_science
+description: The following are multiple choice questions (with answers) about high
+  school computer science.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'Which of the following is an example of the use of a device on the Internet
+      of Things (IoT) ?
+
+      (A) A car alerts a driver that it is about to hit an object. (B) A hiker uses
+      a G P S watch to keep track of her position. (C) A refrigerator orders milk
+      from an online delivery service when the milk in the refrigerator is almost
+      gone. (D) A runner uses a watch with optical sensors to monitor his heart rate.'
+    target: Let's think step by step. The term Internet of Things (IoT) refers to
+      common devices which are connected to the internet, enabling new functionality.
+      Choice A is incorrect because it does not describe an internet connected device.
+      In choice B, the watch is only described as having GPS functionality but no
+      internet connectivity. Choice C describes a common device (a refrigerator) which
+      has internet connectivity enabling new functionality (online ordering). Choice
+      D does not mention internet connectivity for the watch, only optical sensors.
+      The answer is (C).
+  - question: 'Many Web browsers allow users to open anonymous windows. During a browsing
+      session in an anonymous window, the browser does not record a browsing history
+      or a list of downloaded files. When the anonymous window is exited, cookies
+      created during the session are deleted. Which of the following statements about
+      browsing sessions in an anonymous window is true?
+
+      (A) The activities of a user browsing in an anonymous window will not be visible
+      to people who monitor the user''s network, such as the system administrator.
+      (B) Items placed in a Web store''s shopping cart for future purchase during
+      the anonymous browsing session will not be saved on the user''s computer. (C)
+      A user will not be able to log in to e-mail or social media accounts during
+      the anonymous browsing session. (D) A user browsing in an anonymous window will
+      be protected from viruses launched from any web sites visited or files downloaded.'
+    target: "Let's think step by step. Choice A is incorrect as it only describes\
+      \ network traffic, which an anonymous browser does not change. Choice B is correct\
+      \ as it correctly describes how an anonymous browser will prevent saving data\
+      \ on the user\u2019s computer after the session is ended. Choice C is incorrect\
+      \ because an anonymous browser will not prevent logging in to email or social\
+      \ media accounts. Choice D is incorrect because an anonymous browser in itself\
+      \ performs no virus protection. The answer is (B)."
+  - question: "In the program below, the initial value of X is 5 and the initial value\
+      \ of Y is 10.\nIF (X < 0){\n DISPLAY (\"Foxtrot\")\n} ELSE {\n IF (X > Y){\n\
+      \  DISPLAY (\"Hotel\")\n } ELSE {\n  IF (Y > 0){\n   DISPLAY (\"November\")\n\
+      \  } ELSE {\n   DISPLAY (\"Yankee\")\n  }\n }\n}\nWhat is displayed as a result\
+      \ of running the program?\n(A) Foxtrot (B) Hotel (C) November (D) Yankee"
+    target: Let's think step by step. Because X has the value 5, the first conditional
+      IF (X < 0) is false, so we move to the first ELSE clause. Because X is 5 and
+      Y is 10, the second conditional IF (X > Y) is false, so we move to the following
+      ELSE clause. Since Y is 10, the conditional IF (Y > 0) is true, so the command
+      DISPLAY ("November") is executed. The answer is (C).
+  - question: 'What is the output of "abc"[::-1] in Python 3?
+
+      (A) Error (B) abc (C) cba (D) c'
+    target: Let's think step by step. We know that the slicing operator [::-1] takes
+      all of the elements in the string in reverse order, so we reverse the order
+      of the string "abc", resulting in "cba". The answer is (C).
+  - question: "A list of numbers has n elements, indexed from 1 to n. The following algorithm\
+      \ is intended to display the number of elements in the list that have a value\
+      \ greater than 100. The algorithm uses the variables count and position. Steps\
+      \ 3 and 4 are missing.\n Step 1: Set count to 0 and position to 1.\n Step 2:\
+      \ If the value of the element at index position is greater than 100, increase\
+      \ the value of count by 1.\n Step 3: (missing step)\n Step 4: (missing step)\n\
+      \ Step 5: Display the value of count.\nWhich of the following could be used\
+      \ to replace steps 3 and 4 so that the algorithm works as intended?\n(A) Step\
+      \ 3: Increase the value of position by 1.\n  Step 4: Repeat steps 2 and 3 until\
+      \ the value of count is greater than 100.\n(B) Step 3: Increase the value of\
+      \ position by 1.\n  Step 4: Repeat steps 2 and 3 until the value of position\
+      \ is greater than n.\n(C) Step 3: Repeat step 2 until the value of count is\
+      \ greater than 100.\n  Step 4: Increase the value of position by 1.\n(D) Step\
+      \ 3: Repeat step 2 until the value of position is greater than n.\n  Step 4:\
+      \ Increase the value of count by 1."
+    target: 'Let''s think step by step. Choice A is incorrect, because its Step 4
+      has an incorrect termination condition, stopping when count is greater than
+      100. We need to stop after inspecting all elements in the list. Choice B is
+      correct because it correctly increments both count and position, and correctly
+      repeats these steps and terminates when all elements in the list have been inspected.
+      Choice C is incorrect because it incorrectly increments the variable count until
+      its value is greater than 100, regardless of the elements in the list. Choice
+      D is incorrect because its step 3 does not increment the value of position,
+      so it will repeat forever. The answer is (B).'
+tag: mmlu_flan_cot_fewshot_stem
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_computer_science

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_european_history.yaml

@@ -0,0 +1,199 @@
+dataset_name: high_school_european_history
+description: The following are multiple choice questions (with answers) about high
+  school european history.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'This question refers to the following information.
+
+      Albeit the king''s Majesty justly and rightfully is and ought to be the supreme
+      head of the Church of England, and so is recognized by the clergy of this realm
+      in their convocations, yet nevertheless, for corroboration and confirmation
+      thereof, and for increase of virtue in Christ''s religion within this realm
+      of England, and to repress and extirpate all errors, heresies, and other enormities
+      and abuses heretofore used in the same, be it enacted, by authority of this
+      present Parliament, that the king, our sovereign lord, his heirs and successors,
+      kings of this realm, shall be taken, accepted, and reputed the only supreme
+      head in earth of the Church of England, called Anglicans Ecclesia; and shall
+      have and enjoy, annexed and united to the imperial crown of this realm, as well
+      the title and style thereof, as all honors, dignities, preeminences, jurisdictions,
+      privileges, authorities, immunities, profits, and commodities to the said dignity
+      of the supreme head of the same Church belonging and appertaining; and that
+      our said sovereign lord, his heirs and successors, kings of this realm, shall
+      have full power and authority from time to time to visit, repress, redress,
+      record, order, correct, restrain, and amend all such errors, heresies, abuses,
+      offenses, contempts, and enormities, whatsoever they be, which by any manner
+      of spiritual authority or jurisdiction ought or may lawfully be reformed, repressed,
+      ordered, redressed, corrected, restrained, or amended, most to the pleasure
+      of Almighty God, the increase of virtue in Christ''s religion, and for the conservation
+      of the peace, unity, and tranquility of this realm; any usage, foreign land,
+      foreign authority, prescription, or any other thing or things to the contrary
+      hereof notwithstanding.
+
+      English Parliament, Act of Supremacy, 1534
+
+      From the passage, one may infer that the English Parliament wished to argue
+      that the Act of Supremacy would
+
+      (A) give the English king a new position of authority (B) give the position
+      of head of the Church of England to Henry VIII alone and exclude his heirs (C)
+      establish Calvinism as the one true theology in England (D) end various forms
+      of corruption plaguing the Church in England'
+    target: Let's think step by step. We refer to Wikipedia articles on european history
+      for help. The Act of Supremacy states that it grants authority to the king "to
+      repress and extirpate all errors, heresies, and other enormities and abuses",
+      referring to the corruption in the Church of England. The answer is (D).
+  - question: "This question refers to the following information.\nRead the following\
+      \ excerpt.\nThe revolutionary seed had penetrated into every country and spread\
+      \ more or less. It was greatly developed under the r\xE9gime of the military\
+      \ despotism of Bonaparte. His conquests displaced a number of laws, institutions,\
+      \ and customs; broke through bonds sacred among all nations, strong enough to\
+      \ resist time itself; which is more than can be said of certain benefits conferred\
+      \ by these innovators.\nThe monarchs will fulfil the duties imposed upon them\
+      \ by Him who, by entrusting them with power, has charged them to watch over\
+      \ the maintenance of justice, and the rights of all, to avoid the paths of error,\
+      \ and tread firmly in the way of truth. Placed beyond the passions which agitate\
+      \ society, it is in days of trial chiefly that they are called upon to despoil\
+      \ realities of their false appearances, and to show themselves as they are,\
+      \ fathers invested with the authority belonging by right to the heads of families,\
+      \ to prove that, in days of mourning, they know how to be just, wise, and therefore\
+      \ strong, and that they will not abandon the people whom they ought to govern\
+      \ to be the sport of factions, to error and its consequences, which must involve\
+      \ the loss of society.\nUnion between the monarchs is the basis of the policy\
+      \ which must now be followed to save society from total ruin. . . .\nLet them\
+      \ not confound concessions made to parties with the good they ought to do for\
+      \ their people, in modifying, according to their recognized needs, such branches\
+      \ of the administration as require it.\nLet them be just, but strong; beneficent,\
+      \ but strict.\nLet them maintain religious principles in all their purity, and\
+      \ not allow the faith to be attacked and morality interpreted according to the\
+      \ social contract or the visions of foolish sectarians.\nLet them suppress Secret\
+      \ Societies; that gangrene of society.\n\u2014Klemens von Metternich, Political\
+      \ Confession of Faith, 1820\nWhich of the following was the greatest cause of\
+      \ the fears expressed by Metternich in the document above?\n(A) The ideas of\
+      \ personal liberty and nationalism conceived during the Enlightenment resulted\
+      \ in radical revolutions that could spread throughout Europe. (B) The conquest\
+      \ of Europe by Napoleon led to the creation of new factions and shifted the\
+      \ European balance of power. (C) The power of monarchs had grown to the point\
+      \ where it needed to be checked by other powers within each nation or domination\
+      \ of civilians would occur. (D) The rising and falling economic cycle of the\
+      \ newly emerging capitalist economy could lead to civilian unrest that must\
+      \ be suppressed."
+    target: Let's think step by step. We refer to Wikipedia articles on european history
+      for help. The fears of revolution in early 19th century Europe expressed by
+      Klemens von Metternich, a conservative Austrian statesman, were a direct result
+      of the age of Enlightenment, a period of European history where the absolute
+      power of the monarchy was challenged with ideas of individual liberty and nationalism,
+      leading to the French revolution and its effects all over Europe. The answer
+      is (A).
+  - question: 'This question refers to the following information.
+
+      The excerpts below are from the Navigation Acts of 1651.
+
+      [A]fter the first day of December, one thousand six hundred fifty and one, and
+      from thence forwards, no goods or commodities whatsoever of the growth, production
+      or manufacture of Asia, Africa or America, or of any part thereof; or of any
+      islands belonging to them, or which are described or laid down in the usual
+      maps or cards of those places, as well of the English plantations as others,
+      shall be imported or brought into this Commonwealth of England, or into Ireland,
+      or any other lands, islands, plantations, or territories to this Commonwealth
+      belonging, or in their possession, in any other ship or ships, vessel or vessels
+      whatsoever, but only in such as do truly and without fraud belong only to the
+      people of this Commonwealth, or the plantations thereof, as the proprietors
+      or right owners thereof; and whereof the master and mariners are also of the
+      people of this Commonwealth, under the penalty of the forfeiture and loss of
+      all the goods that shall be imported contrary to this act, , , ,
+
+      [N]o goods or commodities of the growth, production, or manufacture of Europe,
+      or of any part thereof, shall after the first day of December, one thousand
+      six hundred fifty and one, be imported or brought into this Commonwealth of
+      England, or any other lands or territories to this Commonwealth belonging, or
+      in their possession, in any ship or ships, vessel or vessels whatsoever, but
+      in such as do truly and without fraud belong only to the people of this Commonwealth,
+      and in no other, except only such foreign ships and vessels as do truly and
+      properly belong to the people of that country or place, of which the said goods
+      are the growth, production or manufacture.
+
+      Which of the following best describes the outcome of the Navigation Acts of
+      1651?
+
+      (A) They served as a catalyst for the growth of English shipping and overseas
+      trade, but did little to limit the prospects of the Dutch in the seventeenth
+      century. (B) They brought about almost immediate hardships for the Dutch economy
+      as their dominance of overseas trade quickly ended. (C) They were rescinded
+      during the restoration of the Stuarts as they sought normal diplomatic relations
+      with the Dutch so not as to need Parliament''s financial support for war. (D)
+      They led to nearly a century of recurrent war between England and the Netherlands,
+      which would not end until after American independence.'
+    target: Let's think step by step. We refer to Wikipedia articles on european history
+      for help. The Navigation Acts of 1651 helped English shipping by restricting
+      the ability of ships from other European countries, especially the Dutch, to
+      transport goods from colonies in Asia and Africa into England. The answer is
+      (A).
+  - question: "This question refers to the following information.\nIn Russia there was\
+      \ nothing going on well, and [Souvarine] was in despair over the news he had\
+      \ received. His old companions were all turning to the politicians; the famous\
+      \ Nihilists who made Europe tremble-sons of village priests, of the lower middle\
+      \ class, of tradesmen-could not rise above the idea of national liberation,\
+      \ and seemed to believe that the world would be delivered-when they had killed\
+      \ their despot&\u2026\n\"Foolery! They'll never get out of it with their foolery.\"\
+      \nThen, lowering his voice still more, in a few bitter words he described his\
+      \ old dream of fraternity. He had renounced his rank and his fortune; he had\
+      \ gone among workmen, only in the hope of seeing at last the foundation of a\
+      \ new society of labour in common. All the sous in his pockets had long gone\
+      \ to the urchins of the settlement; he had been as tender as a brother with\
+      \ the colliers, smiling at their suspicion, winning them over by his quiet workmanlike\
+      \ ways and his dislike of chattering. But decidedly the fusion had not taken\
+      \ place.\nHis voice changed, his eyes grew bright, he fixed them on \xE9tienne,\
+      \ directly addressing him:\n\"Now, do you understand that? These hatworkers\
+      \ at Marseilles who have won the great lottery prize of a hundred thousand francs\
+      \ have gone off at once and invested it, declaring that they are going to live\
+      \ without doing anything! Yes, that is your idea, all of you French workmen;\
+      \ you want to unearth a treasure in order to devour it alone afterwards in some\
+      \ lazy, selfish corner. You may cry out as much as you like against the rich,\
+      \ you haven't got courage enough to give back to the poor the money that luck\
+      \ brings you. You will never be worthy of happiness as long as you own anything,\
+      \ and your hatred of the bourgeois proceeds solely from an angry desire to be\
+      \ bourgeois yourselves in their place.\"\n\xE9mile Zola, French writer, Germinal,\
+      \ 1885\nThe passage displays the direct concern for the welfare of the working\
+      \ classes that was typically a part of which movement?\n(A) Capitalist (B) Scientific\
+      \ (C) Communist (D) Existentialist"
+    target: Let's think step by step. We refer to Wikipedia articles on european history
+      for help. The modern Communist movement aims to establish a classless society
+      based on communal ownership and distribution of property and means of production,
+      thereby especially benefiting the working classes. The answer is (C).
+  - question: "This question refers to the following information.\nThe following excerpt\
+      \ is from a pamphlet.\nYou will do me the justice to remember, that I have always\
+      \ strenuously supported the Right of every man to his own opinion, however different\
+      \ that opinion might be to mine. He who denies to another this right, makes\
+      \ a slave of himself to his present opinion, because he precludes himself the\
+      \ right of changing it.\nThe most formidable weapon against errors of every\
+      \ kind is Reason. I have never used any other, and I trust I never shall.\n\
+      The circumstance that has now taken place in France of the total abolition of\
+      \ the whole national order of priesthood, and of everything appertaining to\
+      \ compulsive systems of religion, and compulsive articles of faith, has not\
+      \ only precipitated my intention, but rendered a work of this kind exceedingly\
+      \ necessary, lest in the general wreck of superstition, of false systems of\
+      \ government, and false theology, we lose sight of morality, of humanity, and\
+      \ of the theology that is true.\nI believe in one God, and no more; and I hope\
+      \ for happiness beyond this life.\nI believe in the equality of man; and I believe\
+      \ that religious duties consist in doing justice, loving mercy, and endeavoring\
+      \ to make our fellow-creatures happy.\nI do not believe in the creed professed\
+      \ by the Jewish church, by the Roman church, by the Greek church, by the Turkish\
+      \ church, by the Protestant church, nor by any church that I know of. My own\
+      \ mind is my own church.\nAll national institutions of churches, whether Jewish,\
+      \ Christian or Turkish, appear to me no other than human inventions, set up\
+      \ to terrify and enslave mankind, and monopolize power and profit.\nI do not\
+      \ mean by this declaration to condemn those who believe otherwise; they have\
+      \ the same right to their belief as I have to mine.\n\u2014Thomas Paine, The\
+      \ Age of Reason, 1794\u20131795\nWhich of the following Enlightenment philosophes\
+      \ designed a system of checks and balances for government to avoid abuses of\
+      \ power?\n(A) Jean Jacques Rousseau (B) Baron Montesquieu (C) Mary Wollstonecraft\
+      \ (D) Adam Smith"
+    target: 'Let''s think step by step. We refer to Wikipedia articles on european
+      history for help. Baron Montesquieu was a 18th centrury French philsopher who
+      wrote extensively against the monoplization of power and advocated for a system
+      of checks and balances in government to prevent the rise of despotism. The answer
+      is (B).'
+tag: mmlu_flan_cot_fewshot_humanities
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_european_history

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_geography.yaml

@@ -0,0 +1,53 @@
+dataset_name: high_school_geography
+description: The following are multiple choice questions (with answers) about high
+  school geography.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'Which one of the following items is an example of nonmaterial culture?
+
+      (A) Dove soap (B) Dove candy bar (C) Dove symbol (D) A dove (bird).'
+    target: Let's think step by step. We refer to Wikipedia articles on geography
+      for help. Nonmaterial culture consists of cultural ideas, beliefs or symbols
+      that are not physical objects. The answer is (C).
+  - question: 'During the third stage of the demographic transition model, which of the
+      following is true?
+
+      (A) Birth rates increase and population growth rate is less rapid. (B) Birth
+      rates decline and population growth rate is less rapid. (C) Birth rates increase
+      and population growth rate increases. (D) Birth rates decrease and population
+      growth rate increases.'
+    target: Let's think step by step. We refer to Wikipedia articles on geography
+      for help. The demographic transition model models the five different stages
+      of population growth as a country goes through economic development, where the
+      third stage refers to a period of declining birth rates and lower population
+      growth. The answer is (B).
+  - question: 'The practice of hiring a foreign third-party service provider to run an
+      operation is called
+
+      (A) outsourcing. (B) offshoring. (C) maquiladoras. (D) locational interdependence.'
+    target: Let's think step by step. We refer to Wikipedia articles on geography
+      for help. "Offshoring" literally means to move or base some of the activities
+      or processes of a company to a foreign country. The answer is (B).
+  - question: 'Which of the following statements is NOT accurate regarding the services
+      provided by local governments in the United States?
+
+      (A) Duplication of efforts occurs often. (B) Social problems of the central
+      city spill over into the surrounding residential suburbs. (C) Inefficiency in
+      providing services occurs often. (D) One neighborhood''s efforts to reduce pollution
+      are always supported by neighboring communities.'
+    target: Let's think step by step. We refer to Wikipedia articles on geography
+      for help. There may be economic, social or political reasons for two neighboring
+      communities and their local governments not agreeing to pollution reduction
+      efforts initiated by one of them. The answer is (D).
+  - question: 'The rate of natural increase of a population is found by subtracting the
+
+      (A) crude death rate from the crude birth date. (B) crude birth rate from the
+      crude death rate. (C) doubling time from the crude birth rate. (D) fertility
+      rate from the crude death rate.'
+    target: 'Let''s think step by step. We refer to Wikipedia articles on geography
+      for help. The difference between number of births and deaths gives the population
+      increase at any given time. The answer is (A).'
+tag: mmlu_flan_cot_fewshot_social_sciences
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_geography

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_government_and_politics.yaml

@@ -0,0 +1,61 @@
+dataset_name: high_school_government_and_politics
+description: The following are multiple choice questions (with answers) about high
+  school government and politics.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'Which of the following best states an argument made by James Madison in
+      The Federalist number 10?
+
+      (A) Honest politicians can prevent factions from developing. (B) Factions are
+      more likely to occur in large republics than in small ones. (C) The negative
+      effects of factionalism can be reduced by a republican government. (D) Free
+      elections are the people''s best defense against factionalism.'
+    target: Let's think step by step. We refer to Wikipedia articles on government
+      and politics for help. In the Federalist number 10, James Madison advocated
+      for a representative republican form of government to guard against factionalism.
+      The answer is (C).
+  - question: 'The term "budget deficit" refers to the
+
+      (A) annual increase in federal spending on the military (B) amount of interest
+      on the national debt (C) difference between the initial budget proposals made
+      by the president and Congress (D) amount the government spends in excess of
+      its revenues'
+    target: Let's think step by step. We refer to Wikipedia articles on government
+      and politics for help. When the goverment spends more than it earns, their difference
+      is the budget deficit. The answer is (D).
+  - question: 'Which of the following statements about cabinet departments is FALSE?
+
+      (A) They are established by the legislative branch. (B) Their members often
+      don''t have much influence over presidential decisions. (C) They cannot all
+      be run by leaders who belong to the same political party the president does.
+      (D) Not every federal agency is a cabinet department.'
+    target: Let's think step by step. We refer to Wikipedia articles on government
+      and politics for help. There is no law stipulating that some cabinet department
+      leaders have to belong to a political party different from that of the president.
+      The answer is (C).
+  - question: 'Which of the following cases established the precedent that a defendant
+      must be informed of the right to remain silent, the right to a lawyer, and protection
+      from self-incrimination?
+
+      (A) Weeks v. United States (B) Betts v. Brady (C) Mapp v. Ohio (D) Miranda v.
+      Arizona'
+    target: Let's think step by step. We refer to Wikipedia articles on government
+      and politics for help. In the landmark Miranda v. Arizona in 1966, the US Supreme
+      Court, based on the Fifth and Sixth Amendment of the US Constitution, guaranteed
+      a defendant's right to an attorney and protection from self-incrimination. The
+      answer is (D).
+  - question: 'Uncertainty over the limits to presidential power is caused primarily
+      by the fact that
+
+      (A) the constitutional definition of those powers is broad and unspecific (B)
+      most people agree that the Constitution places too many limits on presidential
+      power (C) the Supreme Court consistently refuses to rule on cases concerning
+      presidential powers (D) constitutional amendments have greatly increased presidential
+      powers'
+    target: 'Let''s think step by step. We refer to Wikipedia articles on government
+      and politics for help. The US Constitution is not very specific about the powers
+      of the president, leading to uncertainty over its limits. The answer is (A).'
+tag: mmlu_flan_cot_fewshot_social_sciences
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_government_and_politics

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_macroeconomics.yaml

@@ -0,0 +1,53 @@
+dataset_name: high_school_macroeconomics
+description: The following are multiple choice questions (with answers) about high
+  school macroeconomics.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'Which of the following policies best describes supply-side fiscal policy?
+
+      (A) An increase in the money supply (B) Increased government spending (C) Lower
+      taxes on research and development of new technology (D) Higher taxes on household
+      income'
+    target: Let's think step by step. We refer to Wikipedia articles on macroeconomics
+      for help. Supply-side fiscal policy stimulates the economy by encouraging more
+      production of goods and services through reduction in taxes and deregulation.
+      The answer is (C).
+  - question: 'The short-run Phillips curve indicates a
+
+      (A) direct relation between unemployment and inflation (B) direct relation between
+      price and quantity demanded (C) inverse relation between price and quantity
+      demanded (D) inverse relation between unemployment and inflation'
+    target: Let's think step by step. We refer to Wikipedia articles on macroeconomics
+      for help. The short-run Phillips curve shows that whenever unemployment decreases
+      below a natural level, the inflation starts increasing, and vice-versa. The
+      answer is (D).
+  - question: 'Holding all else equal which of the following monetary policies would
+      be used to boost U.S. exports?
+
+      (A) Increasing the discount rate (B) Increasing the reserve ratio (C) Buying
+      government securities (D) Lowering tariffs'
+    target: Let's think step by step. We refer to Wikipedia articles on macroeconomics
+      for help. Buying government securities leads to reduction in demand for US dollars
+      from foreign buyers, thereby making it cheaper and hence making US exports more
+      attractive. The answer is (C).
+  - question: 'A federal deficit occurs when
+
+      (A) exports exceed imports. (B) imports exceed exports. (C) federal tax collections
+      exceed spending. (D) federal spending exceeds federal tax revenues.'
+    target: Let's think step by step. We refer to Wikipedia articles on macroeconomics
+      for help. A federal deficit occurs when federal spending exceeds federal income
+      which is primarily from tax revenues. The answer is (D).
+  - question: 'Which of the following is not included in the U.S. GDP?
+
+      (A) The U.S. military opens a new base in a foreign country with 1000 U.S. personnel.
+      (B) Japanese consumers buy thousands of CDs produced in the United States. (C)
+      An American pop singer performs a sold-out concert in Paris. (D) A French theatrical
+      production tours dozens of American cities.'
+    target: 'Let''s think step by step. We refer to Wikipedia articles on macroeconomics
+      for help. The economic transactions related to the performance of the American
+      pop-singer in Paris happens entirely outside the U.S. and hence is not included
+      in the GDP numbers. The answer is (C).'
+tag: mmlu_flan_cot_fewshot_social_sciences
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_macroeconomics

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_mathematics.yaml

@@ -0,0 +1,51 @@
+dataset_name: high_school_mathematics
+description: The following are multiple choice questions (with answers) about high
+  school mathematics.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'Simplify and write the result with a rational denominator: $$\sqrt{\sqrt[3]{\sqrt{\frac{1}{729}}}}$$
+
+      (A) \frac{3\sqrt{3}}{3} (B) \frac{1}{3} (C) \sqrt{3} (D) \frac{\sqrt{3}}{3}'
+    target: Let's think step by step. Factoring $729=3^6$ and combining the roots
+      $\frac{1}{2}\frac{1}{3}\frac{1}{2}=\frac{1}{12}$, we get that $\sqrt{\sqrt[3]{\sqrt{\frac{1}{729}}}}=\left(\frac{1}{3^6}\right)^{\frac{1}{12}}=\frac{1}{3^{\frac{1}{2}}}=\frac{3}{\sqrt{3}}$
+      The answer is (D).
+  - question: 'Five thousand dollars compounded annually at an $x\%$ interest rate takes
+      six years to double. At the same interest rate, how many years will it take
+      $\$300$ to grow to $\$9600$?
+
+      (A) 12 (B) 1 (C) 30 (D) 5'
+    target: Let's think step by step. To go from $\$300$ to $\$9600$, the value must
+      go up by a factor of $9600/300=32=2^5$. Since at this interest rate it takes
+      six years for it to double, it will take $5*6=30$ years to grow to $\$9600$.
+      The answer is (C).
+  - question: "Ten students take a biology test and receive the following scores: 45,\
+      \ 55, 50, 70, 65, 80, 40, 90, 70, 85. What is the mean of the students\u2019\
+      \ test scores?\n(A) 55 (B) 60 (C) 62 (D) 65"
+    target: Let's think step by step. There are 10 students and the sum of their scores
+      is $45 + 55 + 50 + 70 + 65 + 80 + 40 + 90 + 70 + 85 = 650$, the mean is $650/10=65$.
+      The answer is (D).
+  - question: 'The variable $x$ varies directly as the square of $y$, and $y$ varies
+      directly as the cube of $z$. If $x$ equals $-16$ when $z$ equals 2, what is
+      the value of $x$ when $z$ equals $\frac{1}{2}$?
+
+      (A) -1 (B) 16 (C) -\frac{1}{256} (D) \frac{1}{16}'
+    target: Let's think step by step. We know that $x \propto y^2$ and $y \propto
+      z^3$, so $x = k z^6$ for some constant $k$. Plugging in for $x=-16$ and $z=2$,
+      the constant value is $k=\frac{x}{z^6}=\frac{-16}{64}=-\frac{1}{4}$. So, when
+      $z=\frac{1}{2}$, the value of $x$ is $x=kz^6=-\frac{1}{4}\frac{1}{2^6}=-\frac{1}{256}$.
+      The answer is (C).
+  - question: 'Joe was in charge of lights for a dance. The red light blinks every two
+      seconds, the yellow light every three seconds, and the blue light every five
+      seconds. If we include the very beginning and very end of the dance, how many
+      times during a seven minute dance will all the lights come on at the same time?
+      (Assume that all three lights blink simultaneously at the very beginning of
+      the dance.)
+
+      (A) 3 (B) 15 (C) 6 (D) 5'
+    target: 'Let''s think step by step. The least common multiple of 2, 3 and 5 is
+      30, so during a 7 minute dance, all the three lights will come on at the same
+      time $2*7+1=15$ times. The answer is (B).'
+tag: mmlu_flan_cot_fewshot_stem
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_mathematics

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_microeconomics.yaml

@@ -0,0 +1,56 @@
+dataset_name: high_school_microeconomics
+description: The following are multiple choice questions (with answers) about high
+  school microeconomics.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'Which of the following is necessarily a characteristic of oligopoly?
+
+      (A) Free entry into and exit from the market (B) A few large producers (C) One
+      producer of a good with no close substitutes (D) A homogenous product'
+    target: Let's think step by step. We refer to Wikipedia articles on microeconomics
+      for help. An oligopoly is when a market is dominated by just one or a few number
+      of sellers or producers. To get oligopoly, the market should have high barriers
+      to new entry, and the product has differentiation. The answer is (B).
+  - question: 'If the government subsidizes producers in a perfectly competitive market,
+      then
+
+      (A) the demand for the product will increase (B) the demand for the product
+      will decrease (C) the consumer surplus will increase (D) the consumer surplus
+      will decrease'
+    target: Let's think step by step. We refer to Wikipedia articles on microeconomics
+      for help. (A) and (B) are wrong because the demand curve does not change at
+      all. If the government subsidizes producers, the supply will increase, and thus
+      the consumer surplus also increases. The answer is (C).
+  - question: 'Which of the following is true of a price floor?
+
+      (A) The price floor shifts the demand curve to the left. (B) An effective floor
+      creates a shortage of the good. (C) The price floor shifts the supply curve
+      of the good to the right. (D) To be an effective floor, it must be set above
+      the equilibrium price.'
+    target: Let's think step by step. We refer to Wikipedia articles on microeconomics
+      for help. Price floor does not shift the demand or shift curve. An effective
+      price floor should be set above the equilibrium price, otherwise the market
+      bears and the floor does not have effective effect. The answer is (D).
+  - question: 'The concentration ratio for a monopoly is
+
+      (A) 0 (B) 5 (C) 10 (D) 100'
+    target: Let's think step by step. We refer to Wikipedia articles on microeconomics
+      for help. The concentration ratio is calculated as the sum of market share of
+      a specific number of largest companies. Monopoly means one company or entity
+      controls the entire market, therefore, the concentration ratio is 100 percent.
+      The answer is (D).
+  - question: 'In a competitive labor market for housepainters, which of the following
+      would increase the demand for housepainters?
+
+      (A) An effective minimum wage imposed on this labor market. (B) An increase
+      in the price of gallons of paint. (C) An increase in the construction of new
+      houses. (D) An increase in the price of mechanical painters so long as the output
+      effect exceeds the substitution effect.'
+    target: 'Let''s think step by step. We refer to Wikipedia articles on microeconomics
+      for help. An increase in the construction of new houses means an increase demand
+      of in-house painting, thus increases the demand for housepainters. The answer
+      is (C).'
+tag: mmlu_flan_cot_fewshot_social_sciences
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_microeconomics

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_physics.yaml

@@ -0,0 +1,50 @@
+dataset_name: high_school_physics
+description: The following are multiple choice questions (with answers) about high
+  school physics.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'A microwave oven is connected to an outlet, 120 V, and draws a current
+      of 2 amps. At what rate is energy being used by the microwave oven?
+
+      (A) 10 W (B) 30 W (C) 60 W (D) 240 W'
+    target: Let's think step by step. Rate of energy usage is known as power; in an
+      dissipative electrical circuit, power is given by voltage times current. So
+      in our case, the power is 120 V times 2 amps, or 240 W. The answer is (D).
+  - question: "A point charge, Q = +1 mC, is fixed at the origin. How much work is required\
+      \ to move a charge, Q = +8 \xB5C, from the point (0, 4 meters) to the point\
+      \ (3 meters, 0)?\n(A) 3.5 J (B) 6.0 J (C) 22.5 J (D) 40 J"
+    target: "Let's think step by step. To calculate the work required to move a charge\
+      \ from one location to another in a fixed electric field, it is enough to calculate\
+      \ the potential difference between the two locations. Here, the potential only\
+      \ depends on the distance between the charges; it\u2019s $k q_1 q_2 / r$, where\
+      \ $k$ is Coulomb\u2019s constant. Plugging in values $q_1 = $ 1 mC, $q_2 = 8\
+      \ \\mu$ C, gives the answer as 5.992 J, which rounds to 6 J. The answer is (B)."
+  - question: 'Which of the following conditions will ensure that angular momentum is
+      conserved? I. Conservation of linear momentum II. Zero net external force III.
+      Zero net external torque
+
+      (A) I and II only (B) I and III only (C) II and III only (D) III only'
+    target: Let's think step by step. Torque is defined as the change in angular momentum;
+      if there is zero external torque, angular momentum is conserved. The answer
+      is (D).
+  - question: "A photocell of work function \u03D5 = 2eV is connected to a resistor in\
+      \ series. Light of frequency f = 1 \xD7 10^15 Hz hits a metal plate of the photocell.\
+      \ If the power of the light is P = 100 W, what is the current through the resistor?\n\
+      (A) 2:00 AM (B) 6:00 AM (C) 12:00 AM (D) 24 A"
+    target: Let's think step by step. The only answer above which has units of current
+      is D, 24 A. The answer is (D).
+  - question: "A pipe full of air is closed at one end. A standing wave is produced in\
+      \ the pipe, causing the pipe to sound a note. Which of the following is a correct\
+      \ statement about the wave\u2019s properties at the closed end of the pipe?\n\
+      (A) The pressure is at a node, but the particle displacement is at an antinode.\
+      \ (B) The pressure is at an antinode, but the particle displacement is at a\
+      \ node. (C) The pressure and the particle displacement are both at nodes. (D)\
+      \ The pressure and the particle displacement are both at antinodes."
+    target: 'Let''s think step by step. At the closed end of the pipe, the particles
+      cannot have any net displacement because the pipe closure stops them. So the
+      particle displacement is at a node. This closure also causes the pressure to
+      be maximal, i.e. an antinode. The answer is (B).'
+tag: mmlu_flan_cot_fewshot_stem
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_physics

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_psychology.yaml

@@ -0,0 +1,64 @@
+dataset_name: high_school_psychology
+description: The following are multiple choice questions (with answers) about high
+  school psychology.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'Pascale is interested in the processing strategies children use to learn
+      new information. Pascale would best be classified as what type of psychologist?
+
+      (A) sociocultural (B) clinical (C) cognitive (D) behaviorist'
+    target: Let's think step by step. We refer to Wikipedia articles on psychology
+      for help. Sociocultural psychologist focuses on the effect of societal factors
+      on people. Clinical psychologist focuses on people with mental issues. Cognitive
+      psychologist focuses on how people think and learn, including the processing
+      strategies. Behaviorist focuses more on the environment and experience effect
+      on people. The answer is (C).
+  - question: 'According to Caplan''s model of consultee-centered case consultation,
+      the consultant is primarily interested in
+
+      (A) identifying the causes and solutions of the client''s presenting problems
+      (B) identifying and eliminating the causes of the consultee''s difficulties
+      in handling a problem (C) establishing a hierarchy of authority to enable effective
+      decision making (D) presenting a single, well-defined and unambiguous course
+      of action for the consultant to overcome skills deficits'
+    target: Let's think step by step. We refer to Wikipedia articles on psychology
+      for help. Caplan defines two type of consultation. Client-centered case consultation
+      aims to handle client's problems, while consultee-centered case consultation
+      aims to identify the reason of client's difficulty to solve problems. The answer
+      is (B).
+  - question: 'According to the Individuals with Disabilities Education Improvement Act,
+      which of the following must an educational agency do before it changes the educational
+      placement of a student with a disability?
+
+      (A) Give the child a trial period in the new environment (B) Notify the parents
+      in writing (C) Obtain school board approval (D) Obtain parental consent'
+    target: Let's think step by step. We refer to Wikipedia articles on psychology
+      for help. When the decision to change the educational placement of a student
+      with a disability is made, the educational agency must notify the parents in
+      writing on that date. The answer is (B).
+  - question: 'While swimming in the ocean, Ivan is frightened by a dark shadow in the
+      water even before he has the chance to identify what the shadow is. The synaptic
+      connections taking place during this incident of fright are best described by
+      which of the following?
+
+      (A) Messages are sent from the thalamus directly to the amygdala. (B) Messages
+      are sent from the thalamus to the "what" and "where" pathways. (C) Messages
+      are sent from the parasympathetic nervous system to the cerebral cortex. (D)
+      Messages are sent from the frontal lobes to the pituitary gland.'
+    target: Let's think step by step. We refer to Wikipedia articles on psychology
+      for help. Our neural system has a mechanism that can respond immediate emotional
+      signal before going to the thought center. In the Ivan's case, messages travel
+      directly from thalamus to amygdala. The answer is (A).
+  - question: 'Ani believes that her attitudes and behavior play a central role in what
+      happens to her. Such a belief is likely to be associated with
+
+      (A) a strong superego. (B) low self-esteem. (C) low self-efficacy. (D) an internal
+      locus of control.'
+    target: 'Let''s think step by step. We refer to Wikipedia articles on psychology
+      for help. People with an external locus of control believes fate and luck play
+      an important role in their lives, while people with an internal locus of control
+      believes they control their lives. The answer is (D).'
+tag: mmlu_flan_cot_fewshot_social_sciences
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_psychology

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_statistics.yaml

@@ -0,0 +1,81 @@
+dataset_name: high_school_statistics
+description: The following are multiple choice questions (with answers) about high
+  school statistics.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: 'A new smartwatch is manufactured in one part of a factory, then secured
+      for shipping in another, independent part of the factory. The weight of the
+      smartwatch has a mean of 62 grams and a standard deviation of 1.0 grams. The
+      weight of the packaging (box, user''s guide, bubble wrap, etc.) has a mean of
+      456 grams and a standard deviation of 6 grams. Together, the distribution of
+      the weight of the smartwatch and its packaging would have the following mean
+      and standard deviation:
+
+      (A) Mean 518 grams; standard deviation 7.0 grams (B) Mean 518 grams; standard
+      deviation 3.5 grams (C) Mean 518 grams; standard deviation 6.1 grams (D) Mean
+      394 grams; standard deviation 6.1 grams'
+    target: Let's think step by step. Since the weight of the watch and the weight
+      of the packaging are independent random variables, the mean and variance of
+      their sum is equal to the sum of their individual means and variances. So the
+      mean is 62 + 456 = 518 grams, and the variances is 1.0^2 + 6.0^2 = 37, leading
+      to a standard deviation of 6.1 grams. The answer is (C).
+  - question: 'After a frost warning was issued, the owner of a large orange grove asked
+      his workers to spray all his trees with water. The water was supposed to freeze
+      and form a protective covering of ice around the orange blossom. Nevertheless,
+      the owner suspected that some trees suffered considerable damage due to the
+      frost. To estimate the proportion of trees that suffered more than 50 percent
+      damage due to the frost, he took a random sample of 100 trees from his grove.
+      What is the response variable in this experiment?
+
+      (A) The proportion of trees that suffered more than 50 percent damage due to
+      frost. (B) The number of trees affected by the frost. (C) The number of trees
+      sampled from the grove. (D) For each sampled tree, whether it suffered more
+      than 50 percent damage or at most 50 percent damage.'
+    target: Let's think step by step. In this experiment, the response variable is
+      what is measured. For each tree, what is measured is whether or not it suffered
+      more than 50 percent damage due to the frost. The answer is (D).
+  - question: 'Suppose X and Y are random variables with E(X) = 37, var(X) = 5, E(Y)
+      = 62, and var(Y) = 12. What are the expected value and variance of the random
+      variable X + Y?
+
+      (A) E(X + Y) = 99, var(X + Y) = 8.5 (B) E(X + Y) = 99, var(X + Y) = 13 (C) E(X
+      + Y) = 99, var(X + Y) = 17 (D) There is insufficient information to answer this
+      question.'
+    target: Let's think step by step. While means of sums of random variables add
+      (regardless of whether the variables are independent) in order to determine
+      the variance of a sum of random variables, we need to know not just their individual
+      variances but the covariance of the two variables, which is not given in this
+      problem. The answer is (D).
+  - question: 'Which of the following sets has the smallest standard deviation? Which
+      has the largest?
+
+      I: {1,2,3}
+
+      II: {-10,10}
+
+      III: {100}
+
+      (A) I, II (B) II, III (C) III, I (D) III, II'
+    target: Let's think step by step. The variance of distribution I is the expected
+      squared deviation from its mean (which is 2), so the variance is 2/3 . The variance
+      of distribution II is 10^2 (because both elements are 10 away from the mean
+      of zero). The variance of distribution III is 0, since it has a single entry.
+      So distribution III has the smallest standard deviation and distribution II
+      has the largest. The answer is (D).
+  - question: 'Which of the following is a correct statement about correlation?
+
+      (A) If the slope of the regression line is exactly 1, then the correlation is
+      exactly 1. (B) If the correlation is 0, then the slope of the regression line
+      is undefined. (C) Switching which variable is called x and which is called y
+      changes the sign of the correlation. (D) The correlation r is equal to the slope
+      of the regression line when z-scores for the y-variable are plotted against
+      z-scores for the x-variable.'
+    target: 'Let''s think step by step. Statement A is false because the slope of
+      the regression line being exactly 1 can occur even when the two variables are
+      not perfectly correlated. Statement B is false because uncorrelated variables
+      regression lines can have slope zero. Statement C is false because correlation
+      is symmetric in the two random variables. The answer is (D).'
+tag: mmlu_flan_cot_fewshot_stem
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_statistics

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_us_history.yaml

@@ -0,0 +1,156 @@
+dataset_name: high_school_us_history
+description: The following are multiple choice questions (with answers) about high
+  school us history.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: "This question refers to the following information.\nI come not to urge\
+      \ personal claims, nor to seek individual benefits; I appear as the advocate\
+      \ of those who cannot plead their own cause; I come as the friend of those who\
+      \ are deserted, oppressed, and desolate. In the Providence of God, I am the\
+      \ voice of the maniac whose piercing cries from the dreary dungeons of your\
+      \ jails penetrate not your Halls of Legislation. I am the Hope of the poor crazed\
+      \ beings who pine in the cells, and stalls, and cages, and waste rooms of your\
+      \ poor-houses. I am the Revelation of hundreds of wailing, suffering creatures,\
+      \ hidden in your private dwellings, and in pens and cabins\u2014shut out, cut\
+      \ off from all healing influences, from all mind-restoring cares.\u2026 Could\
+      \ their melancholy histories be spread before you as revealed to my grieved\
+      \ spirit during the last three months, how promptly, how earnestly would you\
+      \ search out the most approved means of relief; how trifling, how insignificant,\
+      \ by comparison, would appear the sacrifices you are asked to make; how would\
+      \ a few dimes and dollars, gathered from each citizen, diminish in value as\
+      \ a possession, compared with the certain benefits and vast good to be secured\
+      \ for the suffering insane...by the consecration and application of a sufficient\
+      \ fund to the construction of a suitable hospital.\u2026\n\u2014Dorothea Dix,\
+      \ Memorial Soliciting a State Hospital for the Protection and Cure of the Insane,\n\
+      Submitted to the General Assembly of North Carolina, November 1848\nDorothea\
+      \ Dix can best be compared to whom?\n(A) Abigail Adams (B) Clara Barton (C)\
+      \ Shirley Temple (D) Hillary Clinton"
+    target: Let's think step by step. We refer to Wikipedia articles on us history
+      for help. Both Dorothea Dix and Clara barton are American nurses. The answer
+      is (B).
+  - question: "This question refers to the following information.\n\"As our late Conduct\
+      \ at the Conestoga Manor and Lancaster have occasioned much Speculation & a\
+      \ great diversity of Sentiments in this and neighboring Governments; some vindicating\
+      \ & others condemning it; some charitably alleviating the Crime, & others maliciously\
+      \ painting it in the most odious & detestable Colours, we think it our duty\
+      \ to lay before the Publick, the whole Matter as it appeared, & still appears,\
+      \ to us. . . .\n\"If these things are not sufficient to prove an unjustifiable\
+      \ Attachment in the Quakers to the Indians Savages, a fixed Resolution to befriend\
+      \ them & an utter insensibility to human Distresses, let us consider a few more\
+      \ recent Facts. When we found the last Summer that we were likely to get no\
+      \ Assistance from the Government, some Volunteers went out at our own Expense,\
+      \ determined to drive our Enemies from our Borders; & when we came near to the\
+      \ great Island, we understood that a Number of their Warriors had gone out against\
+      \ our Frontiers. Upon this we returned and came up with them and fought with\
+      \ them at the Munfey Hill where we lost some of our Men & killed some of their\
+      \ Warriors & thereby saved our Frontiers from this Story in another Expedition.\
+      \ But no sooner had we destroyed their Provisions on the great Island, & ruined\
+      \ their trade with the good People at Bethlehem, but these very Indians, who\
+      \ were justly suspected of having murdered our Friends in Northampton County,\
+      \ were by the Influence of some Quakers taken under the Protection of the Government\
+      \ to screen them from the Resentments of the Friends and Relations of the Murdered,\
+      \ & to support them thro the Winter.\"\n\u2014\"Apology of the Paxton Boys\"\
+      \ (pamphlet), 1764 (Note: \"apology\" in this context should be read as an explanation,\
+      \ not an admission of guilt or regret.\nThe sentiments expressed in the explanation\
+      \ above reflect which of the ongoing tensions during the colonial period of\
+      \ American history?\n(A) Tensions between British policies and the aspirations\
+      \ of North American colonists. (B) Tensions between American Indians allied\
+      \ with the French and those allied with the British. (C) Tensions between freed\
+      \ African Americans and white planters. (D) Tensions between backcountry settlers\
+      \ and elites within colonial America."
+    target: Let's think step by step. We refer to Wikipedia articles on us history
+      for help. After the French and Indian War, the Scotch-Irish settlers attacked
+      American Indians. After the attacks on the Conestoga, about 250 Paxton Boys
+      present their grievances to the Pennsylvania legislature. As mentioned in the
+      information, the Paxton Boys cited resentiment at local elites. The answer is
+      (D).
+  - question: "This question refers to the following information.\nOur leaders talk about\
+      \ stopping aggression from the north, but this was a struggle among groups of\
+      \ Vietnamese until we intervened. We seem bent upon saving the Vietnamese from\
+      \ Ho Chi Minh even if we have to kill them and demolish their country to do\
+      \ it. As the native people survey bombed-out villages, women and children burned\
+      \ by napalm, rice crops destroyed and cities overrun with our military personnel,\
+      \ they are doubtless saying secretly of the Vietcong guerillas and of the American\
+      \ forces, \"A plague on both your houses.\" \u2026 Stop the bombing, north and\
+      \ south, end search and destroy offensive sweeps, and confine our military action\
+      \ to holding operations on the ground. Bombing the north has failed to halt\
+      \ or seriously check the flow of troops to the south and may, in fact, have\
+      \ prompted a much greater war effort by Hanoi.\n\u2014Senator George McGovern,\
+      \ \"The Lessons of Vietnam,\" April 25, 1967\nWhich of the following opinions\
+      \ from the 1960s most directly reflects the perspective of George McGovern's\
+      \ speech?\n(A) Americans must maximize their technological edge in Vietnam.\
+      \ (B) American bombing in Vietnam is step by step leading to progress in the\
+      \ war. (C) American bombing in Vietnam is a failure. (D) America must not give\
+      \ in to defeatism about the war in Vietnam."
+    target: Let's think step by step. We refer to Wikipedia articles on us history
+      for help. "Stop the bombing" and "Bombing the north has failed to halt or seriously
+      check the flow of troops to the south" indicate that the perspective of George
+      McGovern's speech is that Amerian bombing in Vietnam is a failure. The answer
+      is (C).
+  - question: "This question refers to the following information.\n\"In the new Code\
+      \ of Laws which I suppose it will be necessary for you to make I desire you\
+      \ would Remember the Ladies, and be more generous and favorable to them than\
+      \ your ancestors. Do not put such unlimited power into the hands of the Husbands.\
+      \ Remember all Men would be tyrants if they could. If particular care and attention\
+      \ is not paid to the Ladies we are determined to foment a Rebellion, and will\
+      \ not hold ourselves bound by any Laws in which we have no voice, or Representation.\"\
+      \nAbigail Adams, in a letter to John Adams, 1776\n\"Special legislation for\
+      \ woman has placed us in a most anomalous position. Women invested with the\
+      \ rights of citizens in one section\u2014voters, jurors, office-holders\u2014\
+      crossing an imaginary line, are subjects in the next. In some States, a married\
+      \ woman may hold property and transact business in her own name; in others,\
+      \ her earnings belong to her husband. In some States, a woman may testify against\
+      \ her husband, sue and be sued in the courts; in others, she has no redress\
+      \ in case of damage to person, property, or character. In case of divorce on\
+      \ account of adultery in the husband, the innocent wife is held to possess no\
+      \ right to children or property, unless by special decree of the court. But\
+      \ in no State of the Union has the wife the right to her own person, or to any\
+      \ part of the joint earnings of the co-partnership during the life of her husband.\
+      \ In some States women may enter the law schools and practice in the courts;\
+      \ in others they are forbidden. In some universities girls enjoy equal educational\
+      \ advantages with boys, while many of the proudest institutions in the land\
+      \ deny them admittance, though the sons of China, Japan and Africa are welcomed\
+      \ there. But the privileges already granted in the several States are by no\
+      \ means secure.\"\nSusan B. Anthony, \"Declaration of Rights for Women,\" July\
+      \ 4, 1876\nThe sentiments expressed in the second excerpt by Susan B. Anthony\
+      \ are most likely in support of\n(A) the Equal Rights Amendment (B) universal\
+      \ suffrage (C) states' rights (D) prohibition"
+    target: Let's think step by step. We refer to Wikipedia articles on us history
+      for help. The above information mentioned that women are in an anomalous position
+      in terms of legislation. Women's earnings do not belong to themselves, or they
+      cannot testify against her husbands. Susan believes women should have equal
+      legal rights as men. The answer is (B).
+  - question: 'This question refers to the following information.
+
+      "Society in every state is a blessing, but government even in its best state
+      is but a necessary evil; in its worst state an intolerable one; for when we
+      suffer, or are exposed to the same miseries by a government, which we might
+      expect in a country without government, our calamity is heightened by reflecting
+      that we furnish the means by which we suffer. Government, like dress, is the
+      badge of lost innocence; the palaces of kings are built on the ruins of the
+      bowers of paradise. For were the impulses of conscience clear, uniform, and
+      irresistibly obeyed, man would need no other lawgiver; but that not being the
+      case, he finds it necessary to surrender up a part of his property to furnish
+      means for the protection of the rest; and this he is induced to do by the same
+      prudence which in every other case advises him out of two evils to choose the
+      least. Wherefore, security being the true design and end of government, it unanswerably
+      follows that whatever form thereof appears most likely to ensure it to us, with
+      the least expense and greatest benefit, is preferable to all others."
+
+      Thomas Paine, Common Sense, 1776
+
+      Which of the following "miseries" alluded to above were most condemned by Anti-Federalists
+      of the post-Revolutionary era?
+
+      (A) Organized response to Bacon''s Rebellion (B) Federal response to Shays''s
+      Rebellion (C) Federal response to the Whiskey Rebellion (D) Federal response
+      to Pontiac''s Rebellion'
+    target: 'Let''s think step by step. We refer to Wikipedia articles on us history
+      for help. Anti-Federalists do not believe centralized government power, and
+      suspect Washington''s military response to Whiskey Rebellion. Bacon''s Rebellion
+      and Pontiac''s Rebellion happen before the Revolution and they can be ruled
+      out. The answer is (C).'
+tag: mmlu_flan_cot_fewshot_humanities
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_us_history

lm-evaluation-harness/lm_eval/tasks/mmlu/flan_cot_fewshot/mmlu_high_school_world_history.yaml

@@ -0,0 +1,100 @@
+dataset_name: high_school_world_history
+description: The following are multiple choice questions (with answers) about high
+  school world history.
+fewshot_config:
+  sampler: first_n
+  samples:
+  - question: "This question refers to the following information.\n\"At least one of\
+      \ the [world's] societies would have to somehow enormously increase its productivity\
+      \ [in order to achieve global hegemony]. That quantum jump would have to be\
+      \ made before the various scientific, technological, agricultural, and industrial\
+      \ revolutions on which our post-quantum-leap world rests. It could only be accomplished\
+      \ by exploiting the ecosystems, mineral resources, and human assets of whole\
+      \ continents outside the lands of the society making the jump. Western Europe\
+      \ did just that by means of its brutality and guns and, more important, by geographical\
+      \ and ecological luck.\"\nCopyright \xA9 2015 Cambridge University Press.\n\
+      Alfred Crosby, historian, Ecological Imperialism, 2004\nThe \"quantum jump\"\
+      \ mentioned in the passage most directly contributed to which of the following\
+      \ developments in the period 1450\u20131750 C.E.?\n(A) A breakdown in trade\
+      \ routes through the collapse of the established state structure (B) An increase\
+      \ in the population of the world through more plentiful supplies of food (C)\
+      \ The spread of Chinese and Indian belief systems across the world (D) An increase\
+      \ in social unrest"
+    target: Let's think step by step. We refer to Wikipedia articles on world history
+      for help. The "quantum jump" mentioned in the passage refers to the conquest
+      of the New World and the Columbian Exchange. Choice (A) and (C) did not happen
+      in history. Choice (C) refers to the human assets. The answer is (B).
+  - question: "This question refers to the following information.\n\"The struggle against\
+      \ neo-colonialism is not aimed at excluding the capital of the developed world\
+      \ from operating in less developed countries. It is aimed at preventing the\
+      \ financial power of the developed countries being used in such a way as to\
+      \ impoverish the less developed.\nNon-alignment, as practiced by Ghana and many\
+      \ other countries, is based on co-operation with all States whether they be\
+      \ capitalist, socialist or have a mixed economy. Such a policy, therefore, involves\
+      \ foreign investment from capitalist countries, but it must be invested in accordance\
+      \ with a national plan drawn up by the government of the non-aligned State with\
+      \ its own interests in mind. The issue is not what return the foreign investor\
+      \ receives on his investments\u2026The question is one of power. A State in\
+      \ the grip of neo-colonialism is not master of its own destiny.\"\nKwame Nkrumah,\
+      \ Neo-Colonialism, 1965\nWhich of the following provides the best context for\
+      \ Nkrumah's writings?\n(A) The Industrial Revolution (B) Decolonization (C)\
+      \ Regional Free Trade Associations (D) Autarky"
+    target: Let's think step by step. We refer to Wikipedia articles on world history
+      for help. The passage expresses a point that the successful fight against neo-colonialism
+      were in danger and the newly independent nations like Ghana may be re-colonized
+      via financial power of the developed countries. The answer is (B).
+  - question: "This question refers to the following information.\n\"Indeed, as both\
+      \ the fatwas of distinguished [scholars] who base their opinion on reason and\
+      \ tradition alike and the consensus of the Sunni community agree that the ancient\
+      \ obligation of extirpation, extermination, and expulsion of evil innovation\
+      \ must be the aim of our exalted aspiration, for \"Religious zeal is a victory\
+      \ for the Faith of God the Beneficent\"; then, in accordance with the words\
+      \ of the Prophet (Peace upon him!) \"Whosoever introduces evil innovation into\
+      \ our order must be expelled\" and \"Whosoever does aught against our order\
+      \ must be expelled,\" action has become necessary and exigent\u2026\"\nLetter\
+      \ from Ottoman Sultan Selim I to Safavid Shah Ismail I, 1514\nThe letter from\
+      \ Selim I is most clearly an example of which of the following?\n(A) The maintenance\
+      \ of military supremacy at all costs (B) Expanding tensions between religious\
+      \ sects (C) Factors that brought about the collapse of the Ottoman Empire (D)\
+      \ Peacemaking efforts among the Islamic empires"
+    target: Let's think step by step. We refer to Wikipedia articles on world history
+      for help. The passage is an example of expanding tensions between Selim and
+      Ismail. In the passage the Selim references the fatwa and the consensus of the
+      Sunni community to against whosoever introduces evil. The answer is (B).
+  - question: 'This question refers to the following information.
+
+      "The real grievance of the worker is the insecurity of his existence; he is
+      not sure that he will always have work, he is not sure that he will always be
+      healthy, and he foresees that he will one day be old and unfit to work. If he
+      falls into poverty, even if only through a prolonged illness, he is then completely
+      helpless, exam_ins to his own devices, and society does not currently recognize
+      any real obligation towards him beyond the usual help for the poor, even if
+      he has been working all the time ever so faithfully and diligently. The usual
+      help for the poor, however, leaves a lot to be desired, especially in large
+      cities, where it is very much worse than in the country."
+
+      Otto von Bismarck, 1884
+
+      Otto von Bismarck likely made this speech in reaction to which of the following
+      issues?
+
+      (A) Social acceptance of child labor (B) Declining life expectancy in Germany
+      (C) Criticisms of German trade tariffs (D) Negative effects attributed to industrial
+      capitalism'
+    target: Let's think step by step. We refer to Wikipedia articles on world history
+      for help. The passage talks about the grievance of the work under the industrial
+      capitalism. The answer is (D).
+  - question: "This question refers to the following information.\nHe contains all works\
+      \ and desires and all perfumes and all tastes. He enfolds the whole universe\
+      \ and in silence is loving to all. This is the Spirit that is in my heart, this\
+      \ is Brahman. To him I shall come when I go beyond this life, and to him will\
+      \ come he who has faith and doubts not.\n\u2014The Upanishads, India, c. 1000\
+      \ BCE\nTo which religion does the speaker most likely belong?\n(A) Hinduism\
+      \ (B) Buddhism (C) Shintoism (D) Zoroastrianism"
+    target: 'Let''s think step by step. We refer to Wikipedia articles on world history
+      for help. Brahman refers to the ultimate reality of all things in the Hindu
+      religion. In contrast, Buddhism does not have a concept of supreme God. The
+      answer is (A).'
+tag: mmlu_flan_cot_fewshot_humanities
+include: _mmlu_flan_cot_fewshot_template_yaml
+task: mmlu_flan_cot_fewshot_high_school_world_history