Add files using upload-large-folder tool

lm-evaluation-harness/lm_eval/tasks/acpbench/mcq_cot_2shot/prog.yaml

@@ -0,0 +1,12 @@
+task: acp_prog_mcq
+dataset_name: acp_prog_mcq
+include: _mcq_cot_2shot_yaml
+fewshot_config:
+  sampler: first_n
+  samples:
+  - context: 'This is a ferry domain, where the task is to transport cars from their start to their goal locations, using a ferry. Each location is accessible by ferry from each other location. The cars can be debarked or boarded, and the ferry can carry only one car at a time.  There are 2 locations and 2 cars, numbered consecutively.  Currently, the ferry is at l1, with the car c1 on board. The cars are at locations as follows: c0 is at l1.'
+    question: 'Which the following facts hold after performing the action \"travel by sea from location l1 to location l0\" in the current state? **Possible Answers**: A. Car c0 is at location l1 and The ferry is at l1 location.  B. The ferry is at l0 location and The ferry is at l1 location.  C. The ferry is at l0 location.  D. The ferry is at l0 location and Car c0 is at location l1.'
+    answer: "Let's think step by step.   Step 1: The following fact(s) do not hold in the current state: The ferry is at l0 location. Step 2: The action adds the following fact(s): The ferry is at l0 location Step 3: The following fact(s) hold in the current state: Car c0 is at location l1. Step 4: The action deletes the following fact(s): The ferry is at l1 location Step 5: Fact(s) \"The ferry is at l0 location\" are added and Fact(s) \"Car c0 is at location l1\" are not deleted. **Final Answer**: D."
+  - context: 'There are several cities, each containing several locations, some of which are airports. There are also trucks, which can drive within a single city, and airplanes, which can fly between airports. The goal is to get some packages from various locations to various new locations.  There are 2 trucks and 1 airplane, as well as 4 packages. There are 4 locations across 2 cities.  The locations are in cities as follows: l1-1 and l1-0 are in c1; l0-1 and l0-0 are in c0.  Currently, a0 is at l0-0, t1 and p0 are at l1-1, t0 is at l0-1, p1 is in t1, p2 and p3 are in a0.'
+    question: 'Which the following facts hold after performing the action \"drive truck t0 from location l0-1 in city c0 to location l0-1 in the same city\" in the current state? A. p3 is in t1.  B. a0 is at l0-0 and p3 is in t1.  C. a0 is at l0-0.  D. None of the above.'
+    answer: "Let's think step by step.   Step 1: The following fact(s) hold in the current state: a0 is at l0-0. Step 2: The action deletes the following fact(s): t0 is at l0-1 Step 3: Fact(s) \"a0 is at l0-0\" are not deleted. **Final Answer**: C."

lm-evaluation-harness/lm_eval/tasks/acpbench/mcq_cot_2shot/val.yaml

@@ -0,0 +1,12 @@
+task: acp_val_mcq
+dataset_name: acp_val_mcq
+include: _mcq_cot_2shot_yaml
+fewshot_config:
+  sampler: first_n
+  samples:
+  - context: 'This is a ferry domain, where the task is to transport cars from their start to their goal locations, using a ferry. Each location is accessible by ferry from each other location. The cars can be debarked or boarded, and the ferry can carry only one car at a time.  There are 2 locations and 2 cars, numbered consecutively.  Currently, the ferry is at l0 location and it is empty. The cars are at locations as follows: c1 and c0 are at l0. The goal is to reach a state where the following facts hold: Car c0 is at location l1 and Car c1 is at location l1.'
+    question: 'Which of the following claims is true with regard to the following sequence of actions \"board the car c0 at the location l0, travel by sea from location l0 to location l1, unload the car c0 from the ferry to location l1, travel by sea from location l1 to location l0, board the car c1 at location l0, sail from location l0 to location l1, debark the car c1 from the ferry to location l1\" and the current state? A.  The sequence is not applicable.  B. The sequence is a plan.  C. The sequence is applicable, but does not achieve the goal.  D. The sequence is not valid.'
+    answer: "Let's think step by step.   Step 1: For a sequence of actions to be a plan, all actions should be valid, applicable in sequence, and achieve the goal. Step 2: The action sequence is applicable and it achieves the goal. **Final Answer**: B."
+  - context: 'There are several cities, each containing several locations, some of which are airports. There are also trucks, which can drive within a single city, and airplanes, which can fly between airports. The goal is to get some packages from various locations to various new locations.  There are 3 trucks and 1 airplane, as well as 4 packages. There are 9 locations across 3 cities.  The locations are in cities as follows: l1-2, l1-0, and l1-1 are in c1; l0-0, l0-1, and l0-2 are in c0; l2-1, l2-2, and l2-0 are in c2.  Currently, p2 and t1 are at l1-2, p3 is at l2-0, t0 and p0 are at l0-2, p1 is at l1-0, a0 is at l0-0, t2 is at l2-2. The goal is to reach a state where the following facts hold: p1 is at l1-0, p3 is at l2-0, p2 is at l0-1, and p0 is at l1-2.'
+    question: 'Which of the following claims is true with regard to the following sequence of actions \"load object p0 into truck t0 at location l0-2, sail the ship t0 into city c0 from location l0-2 in city l0-0, remove the object p0 from the truck t0 and place it on the location l0-0, load the object p0 from location l0-0 onto the airplane a0, fly the airplane a0 from the airport l0-0 to the airport l1-0, remove the object p0 from the airplane a0 and place it on the location l1-0, load object p2 into truck t1 at location l1-2, navigate the truck t1 from its current location l1-2 in city c1 to the new location l1-0 within the same city place the object p0 into the truck t1 at location l1-0 remove the object p2 from the truck t1 and place it on the location l1-0 load the object p2 from location l1-0 onto the airplane a0 fly the airplane a0 from location l1-0 to location l2-0 fly airplane a0 from airport l2-0 to airport l0-0 unload the object p2 from the airplane a0 at location l0-0 place the object p2 into the truck t0 at location l0-0 navigate the truck t0 from its current location l0-0 in city c0 to the new location l0-1 within the same city offload the object p2 from the truck t0 at location l0-1 drive the truck t1 in city c1 from location l1-0 to location l1-2 offload the object p0 from the truck t1 at location l1-2 navigate the truck t2 from its current location l2-2 in city c2 to the new location l2-1 within the same city\" and the current state? A. The sequence is not valid.  B. The sequence is applicable, but does not achieve the goal.  C. The sequence is a plan.  D.  The sequence is not applicable.'
+    answer: "Let's think step by step.   Step 1: For a sequence of actions to be a plan, all actions should be valid, applicable in sequence, and achieve the goal. Step 2: The action \"sail the ship t0 into city c0 from location l0-2 in city l0-0\" is not valid in this problem. **Final Answer**: A."

lm-evaluation-harness/lm_eval/tasks/aexams/_default_template_yaml

@@ -0,0 +1,18 @@
+dataset_path: Hennara/aexams
+test_split: test
+fewshot_split: dev
+fewshot_config:
+  sampler: first_n
+output_type: multiple_choice
+doc_to_text: "{{question.strip()}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nالجواب："
+doc_to_choice: ["A", "B", "C", "D"]
+doc_to_target: "{{['A', 'B', 'C', 'D'].index(answer)}}"
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

lm-evaluation-harness/lm_eval/tasks/aexams/aexams_Physics.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "Physics"
+"description": "قم بالإجابة على مايلي في مجال الفيزياء \n\n"
+"include": "_default_template_yaml"
+"task": "aexams_Physics"

lm-evaluation-harness/lm_eval/tasks/aexams/aexams_Science.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "Science"
+"description": "قم بالإجابة على مايلي في مجال العلوم \n\n"
+"include": "_default_template_yaml"
+"task": "aexams_Science"

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_1/afrimgsm_ibo.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: ibo
+include: afrimgsm_yaml
+task: afrimgsm_ibo_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_1/afrimgsm_kin.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: kin
+include: afrimgsm_yaml
+task: afrimgsm_kin_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_1/afrimgsm_orm.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: orm
+include: afrimgsm_yaml
+task: afrimgsm_orm_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_1/afrimgsm_swa.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: swa
+include: afrimgsm_yaml
+task: afrimgsm_swa_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_1/afrimgsm_wol.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: wol
+include: afrimgsm_yaml
+task: afrimgsm_wol_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_1/afrimgsm_xho.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: xho
+include: afrimgsm_yaml
+task: afrimgsm_xho_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_1/afrimgsm_yaml

@@ -0,0 +1,35 @@
+tag:
+    - afrimgsm_tasks
+    - afrimgsm_tasks_prompt_1
+dataset_path: masakhane/afrimgsm
+dataset_name: null  # Overridden by language-specific config.
+output_type: generate_until
+test_split: test
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+target_delimiter: ""
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+filter_list:
+  - name: remove_whitespace
+    filter:
+      - function: remove_whitespace
+      - function: take_first
+  - filter:
+    - function: regex
+      group_select: -1
+      regex_pattern: (-?[$0-9.,]{2,})|(-?[0-9]+)
+    - function: take_first
+    name: flexible-extract
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+    ignore_case: true
+    ignore_punctuation: true
+metadata:
+  version: 2.0

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_1/afrimgsm_zul.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: zul
+include: afrimgsm_yaml
+task: afrimgsm_zul_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_2/afrimgsm_amh.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: amh
+include: afrimgsm_yaml
+task: afrimgsm_amh_prompt_2

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_4/afrimgsm_ibo.yaml

@@ -0,0 +1,7 @@
+# Generated by utils.py
+dataset_name: ibo
+doc_to_text: "Answer the given question with the appropriate numerical value, ensuring\
+  \ that the response is clear and without any supplementary information. \n\nQuestion:\
+  \ {{question}} \nAnswer: "
+include: afrimgsm_yaml
+task: afrimgsm_ibo_prompt_4

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_4/afrimgsm_sna.yaml

@@ -0,0 +1,7 @@
+# Generated by utils.py
+dataset_name: sna
+doc_to_text: "Answer the given question with the appropriate numerical value, ensuring\
+  \ that the response is clear and without any supplementary information. \n\nQuestion:\
+  \ {{question}} \nAnswer: "
+include: afrimgsm_yaml
+task: afrimgsm_sna_prompt_4

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_4/afrimgsm_zul.yaml

@@ -0,0 +1,7 @@
+# Generated by utils.py
+dataset_name: zul
+doc_to_text: "Answer the given question with the appropriate numerical value, ensuring\
+  \ that the response is clear and without any supplementary information. \n\nQuestion:\
+  \ {{question}} \nAnswer: "
+include: afrimgsm_yaml
+task: afrimgsm_zul_prompt_4

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_5/afrimgsm_lin.yaml

@@ -0,0 +1,7 @@
+# Generated by utils.py
+dataset_name: lin
+doc_to_text: "For mathematical questions provided in Lingala language. Supply the\
+  \ accurate numeric answer to the provided question. \n\nQuestion: {{question}} \n\
+  Answer: "
+include: afrimgsm_yaml
+task: afrimgsm_lin_prompt_5

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_5/afrimgsm_lug.yaml

@@ -0,0 +1,7 @@
+# Generated by utils.py
+dataset_name: lug
+doc_to_text: "For mathematical questions provided in Luganda language. Supply the\
+  \ accurate numeric answer to the provided question. \n\nQuestion: {{question}} \n\
+  Answer: "
+include: afrimgsm_yaml
+task: afrimgsm_lug_prompt_5

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_5/afrimgsm_sot.yaml

@@ -0,0 +1,7 @@
+# Generated by utils.py
+dataset_name: sot
+doc_to_text: "For mathematical questions provided in Sesotho language. Supply the\
+  \ accurate numeric answer to the provided question. \n\nQuestion: {{question}} \n\
+  Answer: "
+include: afrimgsm_yaml
+task: afrimgsm_sot_prompt_5

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_5/afrimgsm_swa.yaml

@@ -0,0 +1,7 @@
+# Generated by utils.py
+dataset_name: swa
+doc_to_text: "For mathematical questions provided in Swahili language. Supply the\
+  \ accurate numeric answer to the provided question. \n\nQuestion: {{question}} \n\
+  Answer: "
+include: afrimgsm_yaml
+task: afrimgsm_swa_prompt_5

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_5/afrimgsm_vai.yaml

@@ -0,0 +1,6 @@
+# Generated by utils.py
+dataset_name: vai
+doc_to_text: "For mathematical questions provided in Vai language. Supply the accurate\
+  \ numeric answer to the provided question. \n\nQuestion: {{question}} \nAnswer: "
+include: afrimgsm_yaml
+task: afrimgsm_vai_prompt_5

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/prompt_5/afrimgsm_yor.yaml

@@ -0,0 +1,6 @@
+# Generated by utils.py
+dataset_name: yor
+doc_to_text: "For mathematical questions provided in Yoruba language. Supply the accurate\
+  \ numeric answer to the provided question. \n\nQuestion: {{question}} \nAnswer: "
+include: afrimgsm_yaml
+task: afrimgsm_yor_prompt_5

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_eng.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: eng
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_eng_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_ewe.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: ewe
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_ewe_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_hau.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: hau
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_hau_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_ibo.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: ibo
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_ibo_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_twi.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: twi
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_twi_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_vai.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: vai
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_vai_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_wol.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: wol
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_wol_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_xho.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: xho
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_xho_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_yaml

@@ -0,0 +1,37 @@
+tag:
+    - afrimgsm_cot_tasks
+    - afrimgsm_cot_tasks_prompt_1
+dataset_path: masakhane/afrimgsm
+dataset_name: null  # Overridden by language-specific config.
+output_type: generate_until
+training_split: train
+test_split: test
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+  - <|eot_id|>
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+    ignore_case: true
+    ignore_punctuation: true
+filter_list:
+  - name: "strict-match"
+    filter:
+      - function: "regex"
+        regex_pattern: "The answer is (\\-?[0-9\\.\\,]+)"
+      - function: "take_first"
+  - filter:
+    - function: regex
+      group_select: -1
+      regex_pattern: (-?[$0-9.,]{2,})|(-?[0-9]+)
+    - function: take_first
+    name: flexible-extract
+metadata:
+  version: 2.0

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_1/afrimgsm_cot_zul.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: zul
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_zul_prompt_1

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_2/afrimgsm_cot_amh.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: amh
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_amh_prompt_2

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_2/afrimgsm_cot_eng.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: eng
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_eng_prompt_2

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_2/afrimgsm_cot_fra.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: fra
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_fra_prompt_2

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_2/afrimgsm_cot_hau.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: hau
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_hau_prompt_2

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_2/afrimgsm_cot_ibo.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: ibo
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_ibo_prompt_2

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_2/afrimgsm_cot_kin.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: kin
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_kin_prompt_2

lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct_cot/prompt_2/afrimgsm_cot_orm.yaml

@@ -0,0 +1,4 @@
+# Generated by utils.py
+dataset_name: orm
+include: afrimgsm_cot_yaml
+task: afrimgsm_cot_orm_prompt_2