GuoruiSong commited on
Commit
fc36eff
·
verified ·
1 Parent(s): 3a42abf

Add files using upload-large-folder tool

Browse files
Files changed (50) hide show
  1. LaViDa/eval/.github/issue_template.md +8 -0
  2. LaViDa/eval/.github/pull_request_template.md +16 -0
  3. LaViDa/eval/.github/workflows/lint.yml +22 -0
  4. LaViDa/eval/.pre-commit-config.yaml +13 -0
  5. LaViDa/eval/lmms_eval/__init__.py +0 -0
  6. LaViDa/eval/lmms_eval/__main__.py +550 -0
  7. LaViDa/eval/lmms_eval/api/__init__.py +0 -0
  8. LaViDa/eval/lmms_eval/api/filter.py +54 -0
  9. LaViDa/eval/lmms_eval/api/group.py +104 -0
  10. LaViDa/eval/lmms_eval/api/instance.py +29 -0
  11. LaViDa/eval/lmms_eval/api/metrics.py +606 -0
  12. LaViDa/eval/lmms_eval/api/model.py +221 -0
  13. LaViDa/eval/lmms_eval/api/registry.py +185 -0
  14. LaViDa/eval/lmms_eval/api/samplers.py +96 -0
  15. LaViDa/eval/lmms_eval/api/task.py +1630 -0
  16. LaViDa/eval/lmms_eval/caching/__init__.py +0 -0
  17. LaViDa/eval/lmms_eval/caching/cache.py +66 -0
  18. LaViDa/eval/lmms_eval/eval_server.py +0 -0
  19. LaViDa/eval/lmms_eval/evaluator.py +654 -0
  20. LaViDa/eval/lmms_eval/evaluator_utils.py +488 -0
  21. LaViDa/eval/lmms_eval/filters/__init__.py +43 -0
  22. LaViDa/eval/lmms_eval/filters/decontamination.py +23 -0
  23. LaViDa/eval/lmms_eval/filters/extraction.py +282 -0
  24. LaViDa/eval/lmms_eval/filters/selection.py +59 -0
  25. LaViDa/eval/lmms_eval/filters/transformation.py +48 -0
  26. LaViDa/eval/lmms_eval/loggers/__init__.py +2 -0
  27. LaViDa/eval/lmms_eval/loggers/evaluation_tracker.py +452 -0
  28. LaViDa/eval/lmms_eval/loggers/utils.py +127 -0
  29. LaViDa/eval/lmms_eval/loggers/wandb_logger.py +315 -0
  30. LaViDa/eval/lmms_eval/logging_utils.py +368 -0
  31. LaViDa/eval/lmms_eval/models/egogpt.py +472 -0
  32. LaViDa/eval/lmms_eval/models/minicpm_v.py +224 -0
  33. LaViDa/eval/lmms_eval/models/moviechat.py +452 -0
  34. LaViDa/eval/lmms_eval/models/video_llava.py +214 -0
  35. LaViDa/eval/lmms_eval/tasks/activitynetqa/utils.py +287 -0
  36. LaViDa/eval/lmms_eval/tasks/perceptiontest/test/perceptiontest_mcppl.yaml +14 -0
  37. LaViDa/eval/lmms_eval/tasks/video_detail_description/_default_template_yaml +13 -0
  38. LaViDa/eval/lmms_eval/tasks/video_detail_description/utils.py +221 -0
  39. LaViDa/eval/lmms_eval/tasks/video_detail_description/video_detail_description.yaml +12 -0
  40. LaViDa/eval/lmms_eval/utils.py +1018 -0
  41. LaViDa/eval/pyproject.toml +161 -0
  42. LaViDa/eval/run_coco_dream.sh +17 -0
  43. LaViDa/scripts/zero2_fused_adamw.json +41 -0
  44. rl_lladav_video_reversal_wo_t_reward/results/all_rewards.csv +62 -0
  45. rl_lladav_video_reversal_wo_t_reward/results/results-eval-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train.txt +0 -0
  46. rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step11.txt +2 -0
  47. rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step23.txt +2 -0
  48. rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step6.txt +2 -0
  49. rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step7.txt +2 -0
  50. rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step9.txt +2 -0
LaViDa/eval/.github/issue_template.md ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ Before you open an issue, please check if a similar issue already exists or has been closed before.
2
+
3
+ ### When you open an issue, please be sure to include the following
4
+
5
+ - [ ] A descriptive title: [xxx] XXXX
6
+ - [ ] A detailed description
7
+
8
+ Thank you for your contributions!
LaViDa/eval/.github/pull_request_template.md ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Before you open a pull-request, please check if a similar issue already exists or has been closed before.
2
+
3
+ ### When you open a pull-request, please be sure to include the following
4
+
5
+ - [ ] A descriptive title: [xxx] XXXX
6
+ - [ ] A detailed description
7
+
8
+ If you meet the lint warnings, you can use following scripts to reformat code.
9
+
10
+ ```sh
11
+ pip install pre-commit
12
+ pre-commit install
13
+ pre-commit run --all-files
14
+ ```
15
+
16
+ Thank you for your contributions!
LaViDa/eval/.github/workflows/lint.yml ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ name: Lint
2
+
3
+ on: [push, pull_request]
4
+
5
+ jobs:
6
+ lint:
7
+ runs-on: ubuntu-latest
8
+ steps:
9
+ - uses: actions/checkout@v3
10
+ with:
11
+ submodules: true
12
+ fetch-depth: 0
13
+ - name: Set up Python
14
+ uses: actions/setup-python@v4
15
+ with:
16
+ python-version: '3.9'
17
+ - name: Install pre-commit hook
18
+ run: |
19
+ python -m pip install pre-commit
20
+ pre-commit install
21
+ - name: Linting
22
+ run: pre-commit run --all-files
LaViDa/eval/.pre-commit-config.yaml ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ repos:
2
+ - repo: https://github.com/psf/black
3
+ rev: 23.12.1
4
+ hooks:
5
+ - id: black
6
+ language_version: python3
7
+ args: ["--line-length=240"]
8
+ - repo: https://github.com/PyCQA/isort
9
+ rev: 5.13.2
10
+ hooks:
11
+ - id: isort
12
+ language_version: python3
13
+ args: ["--profile", "black"]
LaViDa/eval/lmms_eval/__init__.py ADDED
File without changes
LaViDa/eval/lmms_eval/__main__.py ADDED
@@ -0,0 +1,550 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import argparse
2
+ import datetime
3
+ import importlib
4
+ import json
5
+ import os
6
+ import sys
7
+ import traceback
8
+ import warnings
9
+ from functools import partial
10
+
11
+ import numpy as np
12
+ import yaml
13
+
14
+ warnings.simplefilter("ignore", category=DeprecationWarning)
15
+
16
+ import hashlib
17
+ from pathlib import Path
18
+ from typing import Union
19
+
20
+ from accelerate import Accelerator
21
+ from accelerate.utils import InitProcessGroupKwargs
22
+ from loguru import logger as eval_logger
23
+
24
+ from lmms_eval import evaluator, utils
25
+ from lmms_eval.api.registry import ALL_TASKS
26
+ from lmms_eval.evaluator import request_caching_arg_to_dict
27
+ from lmms_eval.loggers import EvaluationTracker, WandbLogger
28
+ from lmms_eval.tasks import TaskManager
29
+ from lmms_eval.utils import (
30
+ handle_non_serializable,
31
+ make_table,
32
+ simple_parse_args_string,
33
+ )
34
+
35
+
36
+ def _int_or_none_list_arg_type(min_len: int, max_len: int, defaults: str, value: str, split_char: str = ","):
37
+ def parse_value(item):
38
+ item = item.strip().lower()
39
+ if item == "none":
40
+ return None
41
+ try:
42
+ return int(item)
43
+ except ValueError:
44
+ raise argparse.ArgumentTypeError(f"{item} is not an integer or None")
45
+
46
+ items = [parse_value(v) for v in value.split(split_char)]
47
+ num_items = len(items)
48
+
49
+ if num_items == 1:
50
+ # Makes downstream handling the same for single and multiple values
51
+ items = items * max_len
52
+ elif num_items < min_len or num_items > max_len:
53
+ raise argparse.ArgumentTypeError(f"Argument requires {max_len} integers or None, separated by '{split_char}'")
54
+ elif num_items != max_len:
55
+ logging.warning(f"Argument requires {max_len} integers or None, separated by '{split_char}'. " "Missing values will be filled with defaults.")
56
+ default_items = [parse_value(v) for v in defaults.split(split_char)]
57
+ items.extend(default_items[num_items:]) # extend items list with missing defaults
58
+
59
+ return items
60
+
61
+
62
+ def check_argument_types(parser: argparse.ArgumentParser):
63
+ """
64
+ Check to make sure all CLI args are typed, raises error if not
65
+ """
66
+ for action in parser._actions:
67
+ if action.dest != "help" and not action.const:
68
+ if action.type is None:
69
+ raise ValueError(f"Argument '{action.dest}' doesn't have a type specified.")
70
+ else:
71
+ continue
72
+
73
+
74
+ def _handle_non_serializable(o):
75
+ if isinstance(o, np.int64) or isinstance(o, np.int32):
76
+ return int(o)
77
+ elif isinstance(o, set):
78
+ return list(o)
79
+ else:
80
+ return str(o)
81
+
82
+
83
+ def parse_eval_args() -> argparse.Namespace:
84
+ parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter)
85
+ parser.add_argument("--config", default="", help="Path to a yaml file specifying all eval arguments, will ignore cli arguments if specified")
86
+ parser.add_argument("--model", default="hf", help="Name of model e.g. `hf`")
87
+ parser.add_argument(
88
+ "--tasks",
89
+ default=None,
90
+ help="To get full list of tasks, use the command lmms-eval --tasks list",
91
+ )
92
+ parser.add_argument(
93
+ "--model_args",
94
+ default="",
95
+ help="String arguments for model, e.g. `pretrained=EleutherAI/pythia-160m,dtype=float32`",
96
+ )
97
+ parser.add_argument(
98
+ "--num_fewshot",
99
+ type=int,
100
+ default=None,
101
+ help="Number of examples in few-shot context",
102
+ )
103
+ parser.add_argument(
104
+ "--batch_size",
105
+ "-b",
106
+ type=str,
107
+ default=1,
108
+ metavar="auto|auto:N|N",
109
+ help="Acceptable values are 'auto', 'auto:N' or N, where N is an integer. Default 1.",
110
+ )
111
+ parser.add_argument(
112
+ "--max_batch_size",
113
+ type=int,
114
+ default=None,
115
+ metavar="N",
116
+ help="Maximal batch size to try with --batch_size auto.",
117
+ )
118
+ parser.add_argument(
119
+ "--device",
120
+ type=str,
121
+ default=None,
122
+ help="Device to use (e.g. cuda, cuda:0, cpu)",
123
+ )
124
+ parser.add_argument(
125
+ "--output_path",
126
+ default=None,
127
+ type=str,
128
+ metavar="= [dir/file.jsonl] [DIR]",
129
+ help="The path to the output file where the result metrics will be saved. If the path is a directory and log_samples is true, the results will be saved in the directory. Else the parent directory will be used.",
130
+ )
131
+ parser.add_argument(
132
+ "--limit",
133
+ type=float,
134
+ default=None,
135
+ help="Limit the number of examples per task. " "If <1, limit is a percentage of the total number of examples.",
136
+ )
137
+ parser.add_argument(
138
+ "--use_cache",
139
+ "-c",
140
+ type=str,
141
+ default=None,
142
+ metavar="DIR",
143
+ help="A path to a sqlite db file for caching model responses. `None` if not caching.",
144
+ )
145
+ parser.add_argument(
146
+ "--cache_requests",
147
+ type=str,
148
+ default=None,
149
+ choices=["true", "refresh", "delete"],
150
+ help="Speed up evaluation by caching the building of dataset requests. `None` if not caching.",
151
+ )
152
+ parser.add_argument(
153
+ "--check_integrity",
154
+ action="store_true",
155
+ help="Whether to run the relevant part of the test suite for the tasks",
156
+ )
157
+ parser.add_argument(
158
+ "--write_out",
159
+ "-w",
160
+ action="store_true",
161
+ default=False,
162
+ help="Prints the prompt for the first few documents.",
163
+ )
164
+ parser.add_argument(
165
+ "--log_samples",
166
+ action="store_true",
167
+ default=False,
168
+ help="If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis",
169
+ )
170
+ parser.add_argument(
171
+ "--wandb_log_samples",
172
+ action="store_true",
173
+ default=False,
174
+ help="If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis to Weights and Biases",
175
+ )
176
+ parser.add_argument(
177
+ "--log_samples_suffix",
178
+ type=str,
179
+ default="model_outputs",
180
+ help="Specify a suffix for the log_samples file name.",
181
+ )
182
+ parser.add_argument(
183
+ "--system_instruction",
184
+ type=str,
185
+ default=None,
186
+ help="System instruction to be used in the prompt",
187
+ )
188
+ parser.add_argument(
189
+ "--apply_chat_template",
190
+ action="store_true",
191
+ default=False,
192
+ help="If True, applies the chat template to the prompt",
193
+ )
194
+ parser.add_argument(
195
+ "--fewshot_as_multiturn",
196
+ action="store_true",
197
+ default=False,
198
+ help="If True, uses the fewshot as a multi-turn conversation",
199
+ )
200
+ parser.add_argument(
201
+ "--show_config",
202
+ action="store_true",
203
+ default=False,
204
+ help="If True, shows the the full config of all tasks at the end of the evaluation.",
205
+ )
206
+ parser.add_argument(
207
+ "--include_path",
208
+ type=str,
209
+ default=None,
210
+ help="Additional path to include if there are external tasks to include.",
211
+ )
212
+ parser.add_argument(
213
+ "--gen_kwargs",
214
+ default="",
215
+ help=("String arguments for model generation on greedy_until tasks," " e.g. `temperature=0,top_k=0,top_p=0`"),
216
+ )
217
+ parser.add_argument(
218
+ "--verbosity",
219
+ type=str,
220
+ default="INFO",
221
+ help="Log error when tasks are not registered.",
222
+ )
223
+ parser.add_argument(
224
+ "--wandb_args",
225
+ default="",
226
+ help="Comma separated string arguments passed to wandb.init, e.g. `project=lmms-eval,job_type=eval",
227
+ )
228
+ parser.add_argument(
229
+ "--timezone",
230
+ default="Asia/Singapore",
231
+ help="Timezone for datetime string, e.g. Asia/Singapore, America/New_York, America/Los_Angeles. You can check the full list via `import pytz; print(pytz.common_timezones)`",
232
+ )
233
+ parser.add_argument(
234
+ "--hf_hub_log_args",
235
+ type=str,
236
+ default="",
237
+ help="Comma separated string arguments passed to Hugging Face Hub's log function, e.g. `hub_results_org=EleutherAI,hub_repo_name=lm-eval-results`",
238
+ )
239
+ parser.add_argument(
240
+ "--predict_only",
241
+ "-x",
242
+ action="store_true",
243
+ default=False,
244
+ help="Use with --log_samples. Only model outputs will be saved and metrics will not be evaluated.",
245
+ )
246
+ default_seed_string = "0,1234,1234,1234"
247
+ parser.add_argument(
248
+ "--seed",
249
+ type=partial(_int_or_none_list_arg_type, 3, 4, default_seed_string),
250
+ default=default_seed_string, # for backward compatibility
251
+ help=(
252
+ "Set seed for python's random, numpy, torch, and fewshot sampling.\n"
253
+ "Accepts a comma-separated list of 4 values for python's random, numpy, torch, and fewshot sampling seeds, "
254
+ "respectively, or a single integer to set the same seed for all four.\n"
255
+ f"The values are either an integer or 'None' to not set the seed. Default is `{default_seed_string}` "
256
+ "(for backward compatibility).\n"
257
+ "E.g. `--seed 0,None,8,52` sets `random.seed(0)`, `torch.manual_seed(8)`, and fewshot sampling seed to 52. "
258
+ "Here numpy's seed is not set since the second value is `None`.\n"
259
+ "E.g, `--seed 42` sets all four seeds to 42."
260
+ ),
261
+ )
262
+ parser.add_argument(
263
+ "--trust_remote_code",
264
+ action="store_true",
265
+ help="Sets trust_remote_code to True to execute code to create HF Datasets from the Hub",
266
+ )
267
+ parser.add_argument("--process_with_media", action="store_true", help="Whether you will process you dataset with audio, image. By default set to False" "In case some benchmarks need to be processed with media, set this flag to True.")
268
+ args = parser.parse_args()
269
+ return args
270
+
271
+
272
+ def cli_evaluate(args: Union[argparse.Namespace, None] = None) -> None:
273
+ if not args:
274
+ args = parse_eval_args()
275
+
276
+ # Check if no arguments were passed after parsing
277
+ if len(sys.argv) == 1:
278
+ print("┌────────────────────────────���──────────────────────────────────────────────────┐")
279
+ print("│ Please provide arguments to evaluate the model. e.g. │")
280
+ print("│ `lmms-eval --model llava --model_path liuhaotian/llava-v1.6-7b --tasks okvqa` │")
281
+ print("│ Use `lmms-eval --help` for more information. │")
282
+ print("└───────────────────────────────────────────────────────────────────────────────┘")
283
+ sys.exit(1)
284
+
285
+ if args.wandb_args:
286
+ if "name" not in args.wandb_args:
287
+ name = f"{args.model}_{args.model_args}_{utils.get_datetime_str(timezone=args.timezone)}"
288
+ name = utils.sanitize_long_string(name)
289
+ args.wandb_args += f",name={name}"
290
+ wandb_logger = WandbLogger(**simple_parse_args_string(args.wandb_args))
291
+
292
+ # reset logger
293
+ eval_logger.remove()
294
+ eval_logger.add(sys.stdout, colorize=True, level=args.verbosity)
295
+ eval_logger.info(f"Verbosity set to {args.verbosity}")
296
+ os.environ["VERBOSITY"] = args.verbosity
297
+ os.environ["TOKENIZERS_PARALLELISM"] = "false"
298
+
299
+ args_list = []
300
+ results_list = []
301
+ if args.config:
302
+ if not os.path.exists(args.config):
303
+ raise ValueError(f"Config file does not exist: {args.config}")
304
+
305
+ with open(args.config, "r") as file:
306
+ config_args = yaml.safe_load(file)
307
+ config_args = [config_args] if type(config_args) != list else config_args
308
+ # multiple configs, create args list first
309
+ for config in config_args:
310
+ args_copy = argparse.Namespace(**vars(args))
311
+ for key, value in config.items():
312
+ setattr(args_copy, key, value)
313
+ args_list.append(args_copy)
314
+ else:
315
+ args_list.append(args)
316
+
317
+ # initialize Accelerator
318
+ kwargs_handler = InitProcessGroupKwargs(timeout=datetime.timedelta(seconds=60000))
319
+ accelerator = Accelerator(kwargs_handlers=[kwargs_handler])
320
+ if accelerator.is_main_process:
321
+ is_main_process = True
322
+ else:
323
+ is_main_process = False
324
+
325
+ for args in args_list:
326
+ try:
327
+ # if is_main_process and args.wandb_args: # thoughtfully we should only init wandb once, instead of multiple ranks to avoid network traffics and unwanted behaviors.
328
+ # wandb_logger = WandbLogger()
329
+
330
+ results, samples = cli_evaluate_single(args)
331
+ results_list.append(results)
332
+
333
+ accelerator.wait_for_everyone()
334
+ if is_main_process and args.wandb_args:
335
+ try:
336
+ wandb_logger.post_init(results)
337
+ wandb_logger.log_eval_result()
338
+ if args.wandb_log_samples and samples is not None:
339
+ wandb_logger.log_eval_samples(samples)
340
+ except Exception as e:
341
+ eval_logger.info(f"Logging to Weights and Biases failed due to {e}")
342
+ # wandb_logger.finish()
343
+
344
+ except Exception as e:
345
+ if args.verbosity == "DEBUG":
346
+ raise e
347
+ else:
348
+ traceback.print_exc()
349
+ eval_logger.error(f"Error during evaluation: {e}. Please set `--verbosity=DEBUG` to get more information.")
350
+ results_list.append(None)
351
+
352
+ for args, results in zip(args_list, results_list):
353
+ # cli_evaluate will return none if the process is not the main process (rank 0)
354
+ if results is not None:
355
+ print(f"{args.model} ({args.model_args}), gen_kwargs: ({args.gen_kwargs}), limit: {args.limit}, num_fewshot: {args.num_fewshot}, " f"batch_size: {args.batch_size}")
356
+ print(make_table(results))
357
+ if "groups" in results:
358
+ print(make_table(results, "groups"))
359
+
360
+ if args.wandb_args:
361
+ wandb_logger.run.finish()
362
+
363
+
364
+ def cli_evaluate_single(args: Union[argparse.Namespace, None] = None) -> None:
365
+ selected_task_list = args.tasks.split(",") if args.tasks else None
366
+
367
+ if args.include_path is not None:
368
+ eval_logger.info(f"Including path: {args.include_path}")
369
+ task_manager = TaskManager(args.verbosity, include_path=args.include_path, model_name=args.model)
370
+
371
+ # update the evaluation tracker args with the output path and the HF token
372
+ if args.output_path:
373
+ args.hf_hub_log_args += f",output_path={args.output_path}"
374
+ if os.environ.get("HF_TOKEN", None):
375
+ args.hf_hub_log_args += f",token={os.environ.get('HF_TOKEN')}"
376
+
377
+ evaluation_tracker_args = simple_parse_args_string(args.hf_hub_log_args)
378
+ eval_logger.info(f"Evaluation tracker args: {evaluation_tracker_args}")
379
+
380
+ evaluation_tracker = EvaluationTracker(**evaluation_tracker_args)
381
+
382
+ if args.predict_only:
383
+ args.log_samples = True
384
+ if (args.log_samples or args.predict_only) and not args.output_path:
385
+ raise ValueError("Specify --output_path if providing --log_samples or --predict_only")
386
+
387
+ if args.fewshot_as_multiturn and args.apply_chat_template is False:
388
+ raise ValueError("If fewshot_as_multiturn is set, apply_chat_template must be set to True.")
389
+
390
+ if (args.num_fewshot is None or args.num_fewshot == 0) and args.fewshot_as_multiturn:
391
+ raise ValueError("If fewshot_as_multiturn is set, num_fewshot must be greater than 0.")
392
+
393
+ if args.include_path is not None:
394
+ eval_logger.info(f"Including path: {args.include_path}")
395
+
396
+ if "push_samples_to_hub" in evaluation_tracker_args and not args.log_samples:
397
+ eval_logger.warning("Pushing samples to the Hub requires --log_samples to be set. Samples will not be pushed to the Hub.")
398
+
399
+ if args.limit:
400
+ eval_logger.warning(" --limit SHOULD ONLY BE USED FOR TESTING." "REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.")
401
+
402
+ if os.environ.get("LMMS_EVAL_PLUGINS", None):
403
+ args.include_path = [args.include_path] if args.include_path else []
404
+ for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
405
+ package_tasks_location = importlib.util.find_spec(f"{plugin}.tasks").submodule_search_locations[0]
406
+ args.include_path.append(package_tasks_location)
407
+
408
+ if args.tasks is None:
409
+ eval_logger.error("Need to specify task to evaluate.")
410
+ sys.exit()
411
+ elif args.tasks == "list":
412
+ eval_logger.info("Available Tasks:\n - {}".format(f"\n - ".join(sorted(task_manager.list_all_tasks()))))
413
+ sys.exit()
414
+ elif args.tasks == "list_groups":
415
+ eval_logger.info(task_manager.list_all_tasks(list_subtasks=False, list_tags=False))
416
+ sys.exit()
417
+ elif args.tasks == "list_tags":
418
+ eval_logger.info(task_manager.list_all_tasks(list_groups=False, list_subtasks=False))
419
+ sys.exit()
420
+ elif args.tasks == "list_subtasks":
421
+ eval_logger.info(task_manager.list_all_tasks(list_groups=False, list_tags=False))
422
+ sys.exit()
423
+ elif args.tasks == "list_with_num":
424
+ log_message = (
425
+ "\n" + "=" * 70 + "\n" + "\n\tYou are trying to check all the numbers in each task." + "\n\tThis action will download the complete dataset." + "\n\tIf the results are not clear initially, call this again." + "\n\n" + "=" * 70
426
+ )
427
+ eval_logger.info(log_message)
428
+ for task_name in sorted(task_manager.list_all_tasks()):
429
+ try:
430
+ task_dict = get_task_dict([task_name], model_name="llava")
431
+ task_obj = task_dict[task_name]
432
+ if type(task_obj) == tuple:
433
+ group, task_obj = task_obj
434
+ if task_obj is None:
435
+ continue
436
+ eval_logger.info(f"\nTask : {task_obj.config.task}\n - #num : {len(task_obj.test_docs()) if task_obj.has_test_docs() else len(task_obj.validation_docs())}")
437
+ except Exception as e:
438
+ eval_logger.debug(f"\nTask : {task_name} fail to load \n Exception : \n {e}")
439
+ sys.exit()
440
+ else:
441
+ if os.path.isdir(args.tasks):
442
+ import glob
443
+
444
+ task_names = []
445
+ yaml_path = os.path.join(args.tasks, "*.yaml")
446
+ for yaml_file in glob.glob(yaml_path):
447
+ config = utils.load_yaml_config(yaml_file)
448
+ task_names.append(config)
449
+ else:
450
+ task_list = args.tasks.split(",")
451
+ task_names = task_manager.match_tasks(task_list)
452
+ for task in [task for task in task_list if task not in task_names]:
453
+ if os.path.isfile(task):
454
+ config = utils.load_yaml_config(task)
455
+ task_names.append(config)
456
+ task_missing = [task for task in task_list if task not in task_names and "*" not in task] # we don't want errors if a wildcard ("*") task name was used
457
+
458
+ if task_missing:
459
+ missing = ", ".join(task_missing)
460
+ eval_logger.error(
461
+ f"Tasks were not found: {missing}\n" f"{utils.SPACING}Try `lmms-eval --tasks list` for list of available tasks",
462
+ )
463
+ raise ValueError(
464
+ f"Tasks not found: {missing}. Try `lmms-eval --tasks {{list_groups,list_subtasks,list_tags,list}}` to list out all available names for task groupings; only (sub)tasks; tags; or all of the above, or pass '--verbosity DEBUG' to troubleshoot task registration issues."
465
+ )
466
+
467
+ eval_logger.info(f"Selected Tasks: {task_names}")
468
+ request_caching_args = request_caching_arg_to_dict(cache_requests=args.cache_requests)
469
+ datetime_str = utils.get_datetime_str(timezone=args.timezone)
470
+
471
+ results = evaluator.simple_evaluate(
472
+ model=args.model,
473
+ model_args=args.model_args,
474
+ tasks=task_names,
475
+ num_fewshot=args.num_fewshot,
476
+ batch_size=args.batch_size,
477
+ max_batch_size=args.max_batch_size,
478
+ device=args.device,
479
+ use_cache=args.use_cache,
480
+ limit=args.limit,
481
+ check_integrity=args.check_integrity,
482
+ write_out=args.write_out,
483
+ log_samples=args.log_samples,
484
+ evaluation_tracker=evaluation_tracker,
485
+ system_instruction=args.system_instruction,
486
+ apply_chat_template=args.apply_chat_template,
487
+ fewshot_as_multiturn=args.fewshot_as_multiturn,
488
+ gen_kwargs=args.gen_kwargs,
489
+ task_manager=task_manager,
490
+ verbosity=args.verbosity,
491
+ predict_only=args.predict_only,
492
+ random_seed=args.seed[0],
493
+ numpy_random_seed=args.seed[1],
494
+ torch_random_seed=args.seed[2],
495
+ fewshot_random_seed=args.seed[3],
496
+ cli_args=args,
497
+ datetime_str=datetime_str,
498
+ **request_caching_args,
499
+ )
500
+
501
+ if results is not None:
502
+ # breakpoint()
503
+ all_metrics = {}
504
+ for task,task_result in results['results'].items():
505
+ metrics ={}
506
+ for metric_key,metric_value in task_result.items():
507
+ if metric_key in ['alias']:
508
+ continue
509
+ if isinstance(metric_value, (int, float)):
510
+ metrics[metric_key] = metric_value
511
+ for k,v in metrics.items():
512
+ all_metrics[f'{task}/{k}'] = v
513
+ # breakpoint()
514
+ if os.environ.get('CONTROL_LOG_DIR'):
515
+ control_log_dir = os.environ.get('CONTROL_LOG_DIR')
516
+ with open(control_log_dir,'w') as f:
517
+ f.write( json.dumps(all_metrics))
518
+
519
+ if args.log_samples:
520
+ samples = results.pop("samples")
521
+ else:
522
+ samples = None
523
+ dumped = json.dumps(results, indent=4, default=_handle_non_serializable)
524
+ if args.show_config:
525
+ print(dumped)
526
+
527
+ batch_sizes = ",".join(map(str, results["config"]["batch_sizes"]))
528
+
529
+ evaluation_tracker.save_results_aggregated(results=results, samples=samples if args.log_samples else None, datetime_str=datetime_str)
530
+
531
+ if args.log_samples:
532
+ for task_name, config in results["configs"].items():
533
+ evaluation_tracker.save_results_samples(task_name=task_name, samples=samples[task_name])
534
+
535
+ if evaluation_tracker.push_results_to_hub or evaluation_tracker.push_samples_to_hub:
536
+ evaluation_tracker.recreate_metadata_card()
537
+
538
+ return results, samples
539
+ return None, None
540
+
541
+
542
+ def print_results(args, results):
543
+ print(f"{args.model} ({args.model_args}),\ngen_kwargs: ({args.gen_kwargs}),\nlimit: {args.limit},\nnum_fewshot: {args.num_fewshot},\nbatch_size: {args.batch_size}")
544
+ print(evaluator.make_table(results))
545
+ if "groups" in results:
546
+ print(evaluator.make_table(results, "groups"))
547
+
548
+
549
+ if __name__ == "__main__":
550
+ cli_evaluate()
LaViDa/eval/lmms_eval/api/__init__.py ADDED
File without changes
LaViDa/eval/lmms_eval/api/filter.py ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from dataclasses import dataclass
2
+ from typing import List
3
+
4
+ from datasets import Dataset
5
+
6
+ from lmms_eval.api.instance import Instance
7
+
8
+
9
+ class Filter:
10
+ """
11
+ Filter classes operate on a per-task level.
12
+ They take all model outputs (`instance.resps` for all `task.instances`)
13
+ across all instances of a task, and perform operations.
14
+ In a single run, one can configure any number of separate filters or lists of filters.
15
+
16
+ """
17
+
18
+ def __init__(self, *args, **kwargs) -> None:
19
+ """
20
+ Can define custom behavior here, if an individual instantiation of a Filter class should have state.
21
+ """
22
+
23
+ def apply(self, resps, docs):
24
+ """
25
+ Defines the operation to perform on a list of the `inst.resps` properties of `Instance` objects.
26
+ Should return the list of (filtered) response lists *in the same order as they were input*, e.g.
27
+ if pass in [<inst.resps for instance 0>, <inst.resps for instance 1>] should return
28
+ [<filtered resps for instance 0>, <filtered resps for instance 1>]
29
+ """
30
+ return resps
31
+
32
+
33
+ @dataclass
34
+ class FilterEnsemble:
35
+ """
36
+ FilterEnsemble creates a pipeline applying multiple filters.
37
+ Its intended usage is to stack multiple post-processing steps in order.
38
+ `task.apply_filters` should use a list of FilterEnsemble classes that it stores, to apply each
39
+ pipeline separately.
40
+ """
41
+
42
+ name: str
43
+ filters: List[Filter]
44
+
45
+ def apply(self, instances: List[Instance], docs: List[Dataset]) -> None:
46
+ resps = [inst.resps for inst in instances] # operate just on the model responses
47
+ for f in self.filters:
48
+ # apply filters in sequence
49
+ resps = f.apply(resps, docs)
50
+
51
+ # add the end results after filtering to filtered_requests of their respective source instances.
52
+ # has key `self.name`: each FilterEnsemble applied in a given run should use a different name.
53
+ for inst, resp in zip(instances, resps):
54
+ inst.filtered_resps[self.name] = resp
LaViDa/eval/lmms_eval/api/group.py ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import abc
2
+ from dataclasses import asdict, dataclass
3
+ from inspect import getsource
4
+ from typing import Any, Callable, List, Optional, Union
5
+
6
+
7
+ @dataclass
8
+ class AggMetricConfig(dict):
9
+ metric: Optional[str] = None
10
+ aggregation: Optional[str] = "mean"
11
+ weight_by_size: Optional[str] = False
12
+ # list of filter names which should be incorporated into the aggregated metric.
13
+ filter_list: Optional[Union[str, list]] = "none"
14
+
15
+ def __post_init__(self):
16
+ if self.aggregation != "mean" and not callable(self.aggregation):
17
+ raise ValueError(f"Currently, 'mean' is the only pre-defined aggregation across groups' subtasks. Got '{self.aggregation}'.")
18
+
19
+ if isinstance(self.filter_list, str):
20
+ self.filter_list = [self.filter_list]
21
+
22
+
23
+ @dataclass
24
+ class GroupConfig(dict):
25
+ group: Optional[str] = None
26
+ group_alias: Optional[str] = None
27
+ task: Optional[Union[str, list]] = None
28
+ aggregate_metric_list: Optional[Union[List[AggMetricConfig], AggMetricConfig, dict]] = None
29
+ metadata: Optional[dict] = None # by default, not used in the code. allows for users to pass arbitrary info to tasks
30
+
31
+ def __getitem__(self, item):
32
+ return getattr(self, item)
33
+
34
+ def __setitem__(self, item, value):
35
+ return setattr(self, item, value)
36
+
37
+ def __post_init__(self):
38
+ if self.aggregate_metric_list is not None:
39
+ if isinstance(self.aggregate_metric_list, dict):
40
+ self.aggregate_metric_list = [self.aggregate_metric_list]
41
+
42
+ self.aggregate_metric_list = [AggMetricConfig(**item) if isinstance(item, dict) else item for item in self.aggregate_metric_list]
43
+
44
+ def to_dict(self, keep_callable: bool = False) -> dict:
45
+ """dumps the current config as a dictionary object, as a printable format.
46
+ null fields will not be printed.
47
+ Used for dumping results alongside full task configuration
48
+
49
+ :return: dict
50
+ A printable dictionary version of the TaskConfig object.
51
+
52
+ # TODO: should any default value in the TaskConfig not be printed?
53
+ """
54
+ cfg_dict = asdict(self)
55
+ # remove values that are `None`
56
+ for k, v in list(cfg_dict.items()):
57
+ if callable(v):
58
+ cfg_dict[k] = self.serialize_function(v, keep_callable=keep_callable)
59
+ return cfg_dict
60
+
61
+ def serialize_function(self, value: Union[Callable, str], keep_callable=False) -> Union[Callable, str]:
62
+ """Serializes a given function or string.
63
+
64
+ If 'keep_callable' is True, the original callable is returned.
65
+ Otherwise, attempts to return the source code of the callable using 'getsource'.
66
+ """
67
+ if keep_callable:
68
+ return value
69
+ else:
70
+ try:
71
+ return getsource(value)
72
+ except (TypeError, OSError):
73
+ return str(value)
74
+
75
+
76
+ class ConfigurableGroup(abc.ABC):
77
+ def __init__(
78
+ self,
79
+ config: Optional[dict] = None,
80
+ ) -> None:
81
+ self._config = GroupConfig(**config)
82
+
83
+ @property
84
+ def group(self):
85
+ return self._config.group
86
+
87
+ @property
88
+ def group_alias(self):
89
+ return self._config.group_alias
90
+
91
+ @property
92
+ def version(self):
93
+ return self._config.version
94
+
95
+ @property
96
+ def config(self):
97
+ return self._config.to_dict()
98
+
99
+ @property
100
+ def group_name(self) -> Any:
101
+ return self._config.group
102
+
103
+ def __repr__(self):
104
+ return f"ConfigurableGroup(group={self.group}," f"group_alias={self.group_alias})"
LaViDa/eval/lmms_eval/api/instance.py ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from dataclasses import dataclass, field
2
+ from typing import Literal, Tuple
3
+
4
+
5
+ @dataclass
6
+ class Instance:
7
+ request_type: Literal["loglikelihood", "generate_until", "generate_until_multi_round"]
8
+ arguments: tuple
9
+ idx: int
10
+ metadata: Tuple[str, int, int] = field(default_factory=lambda: (None, None, None)) # TODO: better typehints here
11
+ resps: list = field(default_factory=list)
12
+ filtered_resps: dict = field(default_factory=dict)
13
+
14
+ # initialized after init
15
+ task_name: str = None
16
+ doc_id: str = None
17
+ repeats: str = None
18
+ doc: dict = None
19
+
20
+ def __post_init__(self) -> None:
21
+ # unpack metadata field
22
+ self.task_name, self.doc_id, self.repeats = self.metadata["task"], self.metadata["doc_id"], self.metadata["repeats"]
23
+
24
+ @property
25
+ def args(self):
26
+ """
27
+ Returns (string,) where `string` is the string to calculate loglikelihood over
28
+ """
29
+ return self.arguments if isinstance(self.arguments, tuple) else (self.arguments,)
LaViDa/eval/lmms_eval/api/metrics.py ADDED
@@ -0,0 +1,606 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # the code is adapted from https://github.com/EleutherAI/lm-evaluation-harness
2
+ import math
3
+ import random
4
+ import re
5
+ import string
6
+ from collections.abc import Iterable
7
+ from typing import List
8
+
9
+ import numpy as np
10
+ import sacrebleu
11
+ from loguru import logger as eval_logger
12
+
13
+ from lmms_eval.api.registry import register_aggregation, register_metric
14
+
15
+
16
+ # Register Aggregations First
17
+ @register_aggregation("bypass")
18
+ def bypass_agg(arr):
19
+ return 999
20
+
21
+
22
+ @register_aggregation("mean")
23
+ def mean(arr):
24
+ return sum(arr) / len(arr)
25
+
26
+
27
+ @register_aggregation("median")
28
+ def median(arr):
29
+ return arr[len(arr) // 2]
30
+
31
+
32
+ # Certain metrics must be calculated across all documents in a benchmark.
33
+ # We use them as aggregation metrics, paired with no-op passthrough metric fns.
34
+ @register_aggregation("perplexity")
35
+ def perplexity(items):
36
+ return math.exp(-mean(items))
37
+
38
+
39
+ @register_aggregation("weighted_perplexity")
40
+ def weighted_perplexity(items):
41
+ return math.exp(-weighted_mean(items))
42
+
43
+
44
+ @register_aggregation("bits_per_byte")
45
+ def bits_per_byte(items):
46
+ return -weighted_mean(items) / math.log(2)
47
+
48
+
49
+ @register_aggregation("f1")
50
+ def f1_score(items):
51
+ from sklearn.metrics import f1_score
52
+
53
+ unzipped_list = list(zip(*items))
54
+ golds = unzipped_list[0]
55
+ preds = unzipped_list[1]
56
+ fscore = f1_score(golds, preds)
57
+
58
+ return np.max(fscore)
59
+
60
+
61
+ @register_aggregation("matthews_corrcoef")
62
+ def matthews_corrcoef(items):
63
+ from sklearn.metrics import matthews_corrcoef
64
+
65
+ unzipped_list = list(zip(*items))
66
+ golds = unzipped_list[0]
67
+ preds = unzipped_list[1]
68
+ return matthews_corrcoef(golds, preds)
69
+
70
+
71
+ @register_aggregation("bleu")
72
+ def bleu(items):
73
+ """The Bilingual Evaluation Understudy Score, or BLEU for short, is a metric
74
+ for evaluating a generated sentence to a reference sentence. It counts matching
75
+ n-grams in the candidate translation to n-grams in the reference text, where
76
+ 1-gram or unigram would be each token and a bigram comparison would be each
77
+ word pair. The comparison is made regardless of word order
78
+ Source: https://machinelearningmastery.com/calculate-bleu-score-for-text-python/
79
+ Paper: https://www.aclweb.org/anthology/P02-1040/
80
+
81
+ Higher is better
82
+ """
83
+ refs = list(zip(*items))[0]
84
+ preds = list(zip(*items))[1]
85
+ refs, preds = _sacreformat(refs, preds)
86
+ return sacrebleu.corpus_bleu(preds, refs).score
87
+
88
+
89
+ @register_aggregation("chrf")
90
+ def chrf(items):
91
+ """chrF++ is a tool for automatic evaluation of machine translation output
92
+ based on character n-gram precision and recall enhanced with word n-grams.
93
+ Source: https://github.com/m-popovic/chrF
94
+ Paper: https://www.aclweb.org/anthology/W15-3049.pdf
95
+
96
+ Higher is better # TODO I think
97
+ """
98
+ refs = list(zip(*items))[0]
99
+ preds = list(zip(*items))[1]
100
+ refs, preds = _sacreformat(refs, preds)
101
+ return sacrebleu.corpus_chrf(preds, refs).score
102
+
103
+
104
+ @register_aggregation("ter")
105
+ def ter(items):
106
+ """Translation Error Rate is an error metric for machine translation that
107
+ measures the number of edits required to change a system output into one
108
+ of the references
109
+ Source: http://www.cs.umd.edu/~snover/tercom/
110
+ Paper: http://mt-archive.info/AMTA-2006-Snover.pdf
111
+
112
+ Lower is better
113
+ """
114
+ refs = list(zip(*items))[0]
115
+ preds = list(zip(*items))[1]
116
+ refs, preds = _sacreformat(refs, preds)
117
+ return sacrebleu.corpus_ter(preds, refs).score
118
+
119
+
120
+ @register_aggregation("brier_score")
121
+ def brier_score(items): # This is a passthrough function
122
+ gold, predictions = list(zip(*items))
123
+ bs, num_class = np.array(predictions).shape
124
+
125
+ gold = list(gold)
126
+ gold_one_hot = np.eye(num_class)[gold]
127
+ return np.mean(np.sum((predictions - gold_one_hot) ** 2, axis=1))
128
+
129
+
130
+ @register_metric(
131
+ metric="brier_score",
132
+ higher_is_better=False,
133
+ output_type=["multiple_choice"],
134
+ aggregation="brier_score",
135
+ )
136
+ def brier_score_fn(items): # This is a passthrough function
137
+ return items
138
+
139
+
140
+ @register_metric(
141
+ metric="acc",
142
+ higher_is_better=True,
143
+ output_type=["loglikelihood", "multiple_choice"],
144
+ aggregation="mean",
145
+ )
146
+ def acc_fn(items): # This is a passthrough function
147
+ return items
148
+
149
+
150
+ @register_metric(
151
+ metric="acc_norm",
152
+ higher_is_better=True,
153
+ output_type=["loglikelihood", "multiple_choice"],
154
+ aggregation="mean",
155
+ )
156
+ def acc_norm_fn(items): # This is a passthrough function
157
+ return items
158
+
159
+
160
+ @register_metric(
161
+ metric="acc_mutual_info",
162
+ higher_is_better=True,
163
+ output_type="multiple_choice",
164
+ aggregation="mean",
165
+ )
166
+ def acc_mutual_info_fn(items): # This is a passthrough function
167
+ return items
168
+
169
+
170
+ ### the code used in the `exact_match_hf_evaluate` function is ported from
171
+ ### https://github.com/huggingface/evaluate/blob/main/metrics/exact_match/exact_match.py
172
+ ### which is under the apache license.
173
+
174
+ # Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
175
+
176
+ # Licensed under the Apache License, Version 2.0 (the "License");
177
+ # you may not use this file except in compliance with the License.
178
+ # You may obtain a copy of the License at
179
+
180
+ # http://www.apache.org/licenses/LICENSE-2.0
181
+
182
+
183
+ # Unless required by applicable law or agreed to in writing, software
184
+ # distributed under the License is distributed on an "AS IS" BASIS,
185
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
186
+ # See the License for the specific language governing permissions and
187
+ # limitations under the License.
188
+ def exact_match_hf_evaluate(
189
+ predictions,
190
+ references,
191
+ regexes_to_ignore=None,
192
+ ignore_case=False,
193
+ ignore_punctuation=False,
194
+ ignore_numbers=False,
195
+ ):
196
+ if regexes_to_ignore is not None:
197
+ for s in regexes_to_ignore:
198
+ predictions = np.array([re.sub(s, "", x) for x in predictions])
199
+ references = np.array([re.sub(s, "", x) for x in references])
200
+ else:
201
+ predictions = np.asarray(predictions)
202
+ references = np.asarray(references)
203
+
204
+ if ignore_case:
205
+ predictions = np.char.lower(predictions)
206
+ references = np.char.lower(references)
207
+
208
+ if ignore_punctuation:
209
+ repl_table = string.punctuation.maketrans("", "", string.punctuation)
210
+ predictions = np.char.translate(predictions, table=repl_table)
211
+ references = np.char.translate(references, table=repl_table)
212
+
213
+ if ignore_numbers:
214
+ repl_table = string.digits.maketrans("", "", string.digits)
215
+ predictions = np.char.translate(predictions, table=repl_table)
216
+ references = np.char.translate(references, table=repl_table)
217
+
218
+ score_list = predictions == references
219
+
220
+ return {"exact_match": np.mean(score_list)}
221
+
222
+
223
+ ###
224
+
225
+
226
+ @register_metric(
227
+ metric="exact_match",
228
+ higher_is_better=True,
229
+ output_type="generate_until",
230
+ aggregation="mean",
231
+ )
232
+ def exact_match_fn(**kwargs):
233
+ return exact_match_hf_evaluate(**kwargs)
234
+
235
+
236
+ @register_metric(
237
+ metric="perplexity",
238
+ higher_is_better=False,
239
+ output_type="loglikelihood",
240
+ aggregation="perplexity",
241
+ )
242
+ def perplexity_fn(items): # This is a passthrough function
243
+ return items
244
+
245
+
246
+ @register_metric(
247
+ metric="word_perplexity",
248
+ higher_is_better=False,
249
+ output_type="loglikelihood_rolling",
250
+ aggregation="weighted_perplexity",
251
+ )
252
+ def word_perplexity_fn(items): # This is a passthrough function
253
+ return items
254
+
255
+
256
+ @register_metric(
257
+ metric="byte_perplexity",
258
+ higher_is_better=False,
259
+ output_type="loglikelihood_rolling",
260
+ aggregation="weighted_perplexity",
261
+ )
262
+ def byte_perplexity_fn(items): # This is a passthrough function
263
+ return items
264
+
265
+
266
+ @register_metric(
267
+ metric="bits_per_byte",
268
+ higher_is_better=False,
269
+ output_type="loglikelihood_rolling",
270
+ aggregation="bits_per_byte",
271
+ )
272
+ def bits_per_byte_fn(items): # This is a passthrough function
273
+ return items
274
+
275
+
276
+ def levenshtein_distance(s1, s2):
277
+ if len(s1) > len(s2):
278
+ s1, s2 = s2, s1
279
+
280
+ distances = range(len(s1) + 1)
281
+ for i2, c2 in enumerate(s2):
282
+ distances_ = [i2 + 1]
283
+ for i1, c1 in enumerate(s1):
284
+ if c1 == c2:
285
+ distances_.append(distances[i1])
286
+ else:
287
+ distances_.append(1 + min((distances[i1], distances[i1 + 1], distances_[-1])))
288
+ distances = distances_
289
+ return distances[-1]
290
+
291
+
292
+ @register_metric(
293
+ metric="anls",
294
+ higher_is_better=True,
295
+ output_type="generate_until",
296
+ aggregation="mean",
297
+ )
298
+ def anls(
299
+ references,
300
+ predictions,
301
+ thresh_hold=0.5,
302
+ ):
303
+ """https://github.com/QwenLM/Qwen-VL/blob/master/eval_mm/infographicsvqa_eval.py"""
304
+ values = []
305
+ # Unwrap predictions if it's a nested list
306
+ pred = predictions[0] if isinstance(predictions[0], str) else predictions[0][0]
307
+
308
+ for answer in references:
309
+ # preprocess both the answers - gt and prediction
310
+ gt_answer = " ".join(answer.strip().lower().split())
311
+ det_answer = " ".join(pred.strip().lower().split())
312
+
313
+ dist = levenshtein_distance(gt_answer, det_answer)
314
+ length = max(len(answer.upper()), len(pred.upper()))
315
+ values.append(0.0 if length == 0 else float(dist) / float(length))
316
+
317
+ question_result = 1 - min(values)
318
+
319
+ if question_result < thresh_hold:
320
+ question_result = 0
321
+ return {"anls": question_result}
322
+
323
+
324
+ def pop_stddev(arr):
325
+ mu = mean(arr)
326
+ return math.sqrt(sum([(x - mu) ** 2 for x in arr]) / len(arr))
327
+
328
+
329
+ def sample_stddev(arr):
330
+ mu = mean(arr)
331
+ return math.sqrt(sum([(x - mu) ** 2 for x in arr]) / (len(arr) - 1))
332
+
333
+
334
+ def mean_stderr(arr):
335
+ return sample_stddev(arr) / math.sqrt(len(arr))
336
+
337
+
338
+ @register_metric(
339
+ metric="bypass",
340
+ higher_is_better=True,
341
+ output_type=["loglikelihood", "multiple_choice", "generate_until", "generate_until_multi_round"],
342
+ aggregation="bypass",
343
+ )
344
+ def bypass(items):
345
+ return items
346
+
347
+
348
+ @register_metric(
349
+ metric="mcc",
350
+ higher_is_better=True,
351
+ output_type="multiple_choice",
352
+ aggregation="matthews_corrcoef",
353
+ )
354
+ def mcc_fn(items): # This is a passthrough function
355
+ return items
356
+
357
+
358
+ @register_metric(
359
+ metric="f1",
360
+ higher_is_better=True,
361
+ output_type="multiple_choice",
362
+ aggregation="f1",
363
+ )
364
+ def f1_fn(items): # This is a passthrough function
365
+ return items
366
+
367
+
368
+ @register_metric(
369
+ metric="bleu",
370
+ higher_is_better=True,
371
+ output_type=["generate_until", "generate_until_multi_round"],
372
+ aggregation="bleu",
373
+ )
374
+ def bleu_fn(items): # This is a passthrough function
375
+ return items
376
+
377
+
378
+ @register_metric(
379
+ metric="chrf",
380
+ higher_is_better=True,
381
+ output_type=["generate_until", "generate_until_multi_round"],
382
+ aggregation="chrf",
383
+ )
384
+ def chrf_fn(items): # This is a passthrough function
385
+ return items
386
+
387
+
388
+ @register_metric(
389
+ metric="ter",
390
+ higher_is_better=True,
391
+ output_type=["generate_until", "generate_until_multi_round"],
392
+ aggregation="ter",
393
+ )
394
+ def ter_fn(items): # This is a passthrough function
395
+ return items
396
+
397
+
398
+ @register_metric(
399
+ metric="acc_all",
400
+ higher_is_better=True,
401
+ output_type="loglikelihood",
402
+ aggregation="mean",
403
+ )
404
+ def acc_all(items):
405
+ # Only count as correct if all answers are labeled correctly for each question
406
+ question_scoring_dict = {}
407
+ preds = list(zip(*items))[0]
408
+ docs = list(zip(*items))[1]
409
+
410
+ for doc, pred in zip(docs, preds):
411
+ paragraph_id = doc["idx"]["paragraph"]
412
+ question_id = doc["idx"]["question"]
413
+ if (paragraph_id, question_id) not in question_scoring_dict:
414
+ question_scoring_dict[(paragraph_id, question_id)] = []
415
+
416
+ gold_label = doc["label"] == 1
417
+
418
+ question_scoring_dict[(paragraph_id, question_id)].append(gold_label == pred)
419
+ acc = np.mean([int(all(x)) for x in question_scoring_dict.values()])
420
+ return acc
421
+
422
+
423
+ def acc_all_stderr(items):
424
+ # Only count as correct if all answers are labeled correctly for each question
425
+ question_scoring_dict = {}
426
+ preds = list(zip(*items))[0]
427
+ docs = list(zip(*items))[1]
428
+
429
+ for doc, pred in zip(docs, preds):
430
+ question_id = doc["idx"]["question"]
431
+ if question_id not in question_scoring_dict:
432
+ question_scoring_dict[question_id] = []
433
+
434
+ gold_label = doc["label"] == 1
435
+ question_scoring_dict[question_id].append(gold_label == pred)
436
+
437
+ acc = mean_stderr([int(all(x)) for x in question_scoring_dict.values()])
438
+ return acc
439
+
440
+
441
+ def metric_max_over_ground_truths(metric_fn, prediction, ground_truths):
442
+ """Compute max metric between prediction and each ground truth."""
443
+ scores_for_ground_truths = []
444
+ for ground_truth in ground_truths:
445
+ score = metric_fn(prediction, ground_truth)
446
+ scores_for_ground_truths.append(score)
447
+ return max(scores_for_ground_truths)
448
+
449
+
450
+ def weighted_mean(items):
451
+ a, b = zip(*items)
452
+ return sum(a) / sum(b)
453
+
454
+
455
+ def is_non_str_iterable(obj):
456
+ return isinstance(obj, Iterable) and not isinstance(obj, str)
457
+
458
+
459
+ def _sacreformat(refs, preds):
460
+ """Format refs and preds for sacrebleu corpus calculation. It is very particular"""
461
+ # Sacrebleu expects (List[str], List[List[str])
462
+ # e.g. sacrebleu.corpus_bleu([pred_t], [[ref1_stream], [ref2_stream], ...])
463
+
464
+ # Note [ref1_stream] is the first reference for each pred.
465
+ # So lists are size N and (M, N) for N preds and M possible refs for each pred
466
+ # This is a different order of dimensions that I would expect
467
+
468
+ # We expect refs to be List[str] or List[List[str]], the outer list corresponding to preds
469
+ # Must become List[List[str]] with the inner list corresponding to preds
470
+ if not is_non_str_iterable(refs):
471
+ refs = list(refs)
472
+ if not is_non_str_iterable(refs[0]):
473
+ refs = [[ref] for ref in refs]
474
+ refs = list(zip(*refs))
475
+ # Note the number of refs in each ref list much match the number of preds
476
+
477
+ # We expect preds to be List[str] or List[List[str]]. Must become List[str]
478
+ if not is_non_str_iterable(preds):
479
+ preds = list(preds)
480
+ if is_non_str_iterable(preds[0]):
481
+ assert len(preds[0]) == 1, f"Pred must be a str, was {preds[0]}"
482
+ preds = [pred[0] for pred in preds]
483
+
484
+ return refs, preds
485
+
486
+
487
+ # stderr stuff
488
+
489
+
490
+ class _bootstrap_internal:
491
+ def __init__(self, f, n) -> None:
492
+ self.f = f
493
+ self.n = n
494
+
495
+ def __call__(self, v):
496
+ i, xs = v
497
+ rnd = random.Random()
498
+ rnd.seed(i)
499
+ res = []
500
+ for _ in range(self.n):
501
+ res.append(self.f(rnd.choices(xs, k=len(xs))))
502
+ return res
503
+
504
+
505
+ def bootstrap_stderr(f, xs, iters):
506
+ import multiprocessing as mp
507
+
508
+ pool = mp.Pool(mp.cpu_count())
509
+ # this gives a biased estimate of the stderr (i.e w/ the mean, it gives something
510
+ # equivalent to stderr calculated without Bessel's correction in the stddev.
511
+ # Unfortunately, I haven't been able to figure out what the right correction is
512
+ # to make the bootstrap unbiased - i considered multiplying by sqrt(n/(n-1)) but
513
+ # that would be ad-hoc and I can't prove that that would actually be an unbiased estimator)
514
+ # Thankfully, shouldn't matter because our samples are pretty big usually anyways
515
+ res = []
516
+ chunk_size = min(1000, iters)
517
+ from tqdm import tqdm
518
+
519
+ print("bootstrapping for stddev:", f.__name__)
520
+ for bootstrap in tqdm(
521
+ pool.imap(
522
+ _bootstrap_internal(f, chunk_size),
523
+ [(i, xs) for i in range(iters // chunk_size)],
524
+ ),
525
+ total=iters // chunk_size,
526
+ ):
527
+ # sample w replacement
528
+ res.extend(bootstrap)
529
+
530
+ pool.close()
531
+ return sample_stddev(res)
532
+
533
+
534
+ def stderr_for_metric(metric, bootstrap_iters: int):
535
+ if bootstrap_iters <= 0:
536
+ # return no function (don't compute stderr) if bootstrap iters = 0
537
+ return None
538
+
539
+ bootstrappable = [
540
+ median,
541
+ matthews_corrcoef,
542
+ f1_score,
543
+ perplexity,
544
+ bleu,
545
+ chrf,
546
+ ter,
547
+ ]
548
+
549
+ if metric in bootstrappable:
550
+ return lambda x: bootstrap_stderr(metric, x, iters=bootstrap_iters)
551
+
552
+ stderr = {mean: mean_stderr, acc_all: acc_all_stderr}
553
+
554
+ return stderr.get(metric, None)
555
+
556
+
557
+ def pooled_sample_stderr(stderrs: List[float], sizes: List[int]):
558
+ # Used to aggregate bootstrapped stderrs across subtasks in a group,
559
+ # when we are weighting by the size of each subtask.
560
+ #
561
+
562
+ assert len(stderrs) == len(sizes)
563
+
564
+ # formula source: https://en.wikipedia.org/wiki/Pooled_variance
565
+ # and: https://stats.stackexchange.com/a/4841331
566
+ # this empirically seems to match running `stderr_for_metric` on all instances
567
+ # from the subtasks concatenated with each other.
568
+ pooled_sample_var = (sum([(size - 1) * stderr**2 * size for size, stderr in zip(sizes, stderrs)])) / (sum(sizes) - len(sizes))
569
+
570
+ return np.sqrt(pooled_sample_var / sum(sizes))
571
+
572
+
573
+ def combined_sample_stderr(stderrs: List[float], sizes: List[int], metrics=None):
574
+ assert metrics is not None, "Need to pass a list of each subtask's metric for this stderr aggregation"
575
+ assert len(stderrs) == len(sizes) and len(sizes) == len(metrics)
576
+
577
+ # See https://github.com/EleutherAI/lm-evaluation-harness/pull/1390 for more documentation.
578
+ # This formula depends on sample means.
579
+ # removed because it seems to give erroneously huge stderrs for groupings of tasks
580
+ # and does not seem to match up with bootstrap-calculated stderrs for groups.
581
+
582
+ ### don't use this unless a statistician has told you it's the right thing to do ###
583
+
584
+ # accumulators: we'll aggregate pairwise N - 1 times
585
+ variance = stderrs[0] ** 2
586
+ curr_size = sizes[0]
587
+ curr_score = metrics[0]
588
+
589
+ for stderr, size, score in zip(stderrs[1:], sizes[1:], metrics[1:]):
590
+ curr_score = ((curr_score * curr_size) + (score * size)) / (curr_size + size) # NOTE: this assumes our aggregation fn is "mean"
591
+
592
+ variance = ((curr_size - 1) * variance + (size - 1) * (stderr**2)) / (curr_size + size - 1) + curr_size * size / ((curr_size + size) * (curr_size + size - 1)) * (curr_score - score) ** 2
593
+
594
+ return np.sqrt(variance)
595
+
596
+
597
+ def aggregate_subtask_metrics(metrics, sizes, weight_by_size=True):
598
+ # A helper function that is used to aggregate
599
+ # subtask scores cross-task.
600
+ # TODO: does not hold for non-mean aggregations
601
+ if not weight_by_size:
602
+ sizes = [1] * len(sizes)
603
+
604
+ assert len(metrics) == len(sizes)
605
+
606
+ return sum([metric * size for metric, size in zip(metrics, sizes)]) / sum(sizes)
LaViDa/eval/lmms_eval/api/model.py ADDED
@@ -0,0 +1,221 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import abc
2
+ import hashlib
3
+ import json
4
+ import os
5
+ from typing import List, Optional, Tuple, Type, TypeVar, Union
6
+
7
+ from loguru import logger as eval_logger
8
+ from sqlitedict import SqliteDict
9
+ from tqdm import tqdm
10
+
11
+ from lmms_eval import utils
12
+ from lmms_eval.api.instance import Instance
13
+
14
+ T = TypeVar("T", bound="lmms")
15
+
16
+
17
+ class lmms(abc.ABC):
18
+ def __init__(self) -> None:
19
+ """Defines the interface that should be implemented by all lmms subclasses.
20
+ lmmss are assumed to take image-text as input and yield strings as output
21
+ (inputs/outputs should be tokenization-agnostic.)
22
+ """
23
+ # set rank and world size to a single process, by default.
24
+ self._rank = 0
25
+ self._world_size = 1
26
+ self.cache_hook = CacheHook(None)
27
+ self.task_dict = {}
28
+
29
+ @abc.abstractmethod
30
+ def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
31
+ """Compute log-likelihood of generating a continuation from a context.
32
+ Downstream tasks should attempt to use loglikelihood instead of other
33
+ LMM calls whenever possible.
34
+
35
+ :param requests: list[Instance]
36
+ A list of Instance objects, with property `args` which returns a tuple (context, continuation).
37
+ `context: str`
38
+ Context string. Implementations of LMM must be able to handle an
39
+ empty context string.
40
+ `continuation: str`
41
+ The continuation over which log likelihood will be calculated. If
42
+ there is a word boundary, the space should be in the continuation.
43
+ For example, context="hello" continuation=" world" is correct.
44
+ 'visual_list: list[dict]'
45
+ Visual input to the model. Can be None.
46
+
47
+ :return: list[tuple[float, bool]]
48
+ A list of pairs (logprob, isgreedy)
49
+ `logprob: float`
50
+ The log probability of `continuation`.
51
+ `isgreedy`:
52
+ Whether `continuation` would be generated by greedy sampling from `context`.
53
+ """
54
+ pass
55
+
56
+ # TODO: Add an optional max length
57
+ @abc.abstractmethod
58
+ def generate_until(self, requests) -> List[str]:
59
+ """Generate greedily until a stopping sequence
60
+
61
+ :param requests: list[Instance]
62
+ A list of Instance objects with property `args` which returns a tuple (context, until).
63
+ context: str
64
+ Context string
65
+ generation_kwargs: dict
66
+ Generation Kwargs
67
+ 'visual_list: list[dict]'
68
+ Visual input to the model. Can be None.
69
+ :return: list[str]
70
+ A list of strings continuation
71
+ continuation: str
72
+ The generated continuation.
73
+ """
74
+ pass
75
+
76
+ @abc.abstractmethod
77
+ def generate_until_multi_round(self, requests) -> List[str]:
78
+ """Generate greedily until a stopping sequence
79
+
80
+ :param requests: list[Instance]
81
+ A list of Instance objects with property `args` which returns a tuple (context, until).
82
+ context: str
83
+ Context string
84
+ generation_kwargs: dict
85
+ Generation Kwargs
86
+ 'visual_list: list[dict]'
87
+ Visual input to the model. Can be None.
88
+ :return: list[str]
89
+ A list of strings continuation
90
+ continuation: str
91
+ The generated continuation.
92
+ """
93
+ pass
94
+
95
+ @classmethod
96
+ def create_from_arg_string(cls: Type[T], arg_string: str, additional_config: Optional[dict] = None) -> T:
97
+ """
98
+ Creates an instance of the LMM class using the given argument string and additional config.
99
+
100
+ Parameters:
101
+ - arg_string: A string containing arguments in the format key1=value1,key2=value2.
102
+ - additional_config: Optional dictionary containing additional configuration parameters.
103
+
104
+ Returns:
105
+ - Instance of the LMM class.
106
+ """
107
+ additional_config = {} if additional_config is None else additional_config
108
+ args = utils.simple_parse_args_string(arg_string)
109
+ args2 = {k: v for k, v in additional_config.items() if v is not None}
110
+ return cls(**args, **args2)
111
+
112
+ @property
113
+ def rank(self):
114
+ # used in the case of parallelism. Hardcoded to
115
+ # ensure no errors arise using API models which do
116
+ # not support multi-device parallelism nor expect it.
117
+ return self._rank
118
+
119
+ @property
120
+ def world_size(self):
121
+ # used in the case of parallelism. Hardcoded to
122
+ # ensure no errors arise using API models which do
123
+ # not support multi-device parallelism nor expect it.
124
+ return self._world_size
125
+
126
+ def set_cache_hook(self, cache_hook) -> None:
127
+ self.cache_hook = cache_hook
128
+
129
+
130
+ ### SQLite-based caching of LMM responses
131
+ def hash_args(attr, args):
132
+ dat = json.dumps([attr] + list(args))
133
+ return hashlib.sha256(dat.encode("utf-8")).hexdigest()
134
+
135
+
136
+ class CacheHook:
137
+ def __init__(self, cachinglm) -> None:
138
+ if cachinglm is None:
139
+ self.dbdict = None
140
+ return
141
+
142
+ self.dbdict = cachinglm.dbdict
143
+
144
+ def add_partial(self, attr, req, res) -> None:
145
+ if self.dbdict is None:
146
+ return
147
+ hsh = hash_args(attr, req)
148
+ self.dbdict[hsh] = res
149
+
150
+
151
+ class CachingLMM:
152
+ def __init__(self, lm, cache_db) -> None:
153
+ """LMM wrapper that returns cached results if they exist, and uses the underlying LMM if not.
154
+
155
+ :param lm: LMM
156
+ Underlying LMM
157
+ :param cache_db: str
158
+ Path to cache db
159
+ """
160
+ self.lm = lm
161
+ self.cache_db = cache_db
162
+ if os.path.dirname(cache_db):
163
+ os.makedirs(os.path.dirname(cache_db), exist_ok=True)
164
+ self.dbdict = SqliteDict(cache_db, autocommit=True)
165
+
166
+ # add hook to lm
167
+ lm.set_cache_hook(self.get_cache_hook())
168
+
169
+ def __getattr__(self, attr):
170
+ lm_attr = getattr(self.lm, attr)
171
+ if not callable(lm_attr):
172
+ return lm_attr
173
+
174
+ def fn(requests):
175
+ res = []
176
+ remaining_reqs = []
177
+ warned = False
178
+ # figure out which ones are cached and which ones are new
179
+ eval_logger.info(f"Loading '{attr}' responses from cache '{self.cache_db}' where possible...")
180
+ for req in tqdm(requests):
181
+ hsh = hash_args(attr, req.args)
182
+ if attr in ["generate_until", "generate_until_multi_round"] and req.args[1].get("do_sample", False):
183
+ # when we are doing non-greedy generation, don't use the cache
184
+ # (else every "randomly sampled" generation would be identical for repeats > 1).
185
+ if not warned:
186
+ eval_logger.warning(f"Arguments to lm.generate_until() '{req.args[1]}' include non-deterministic sampling. Caching will not be performed for such requests.")
187
+ warned = True
188
+ res.append(None)
189
+ remaining_reqs.append(req)
190
+ elif hsh in self.dbdict:
191
+ ob = self.dbdict[hsh]
192
+
193
+ assert ob is not None
194
+
195
+ res.append(ob)
196
+ else:
197
+ res.append(None)
198
+ remaining_reqs.append(req)
199
+
200
+ # actually run the LMM on the requests that do not have cached results
201
+ rem_res = getattr(self.lm, attr)(remaining_reqs)
202
+
203
+ # stick the new ones back into the list and also cache any of the new ones
204
+ resptr = 0
205
+ for req, r in zip(remaining_reqs, rem_res):
206
+ while res[resptr] is not None:
207
+ resptr += 1
208
+
209
+ res[resptr] = r
210
+
211
+ # caching
212
+ hsh = hash_args(attr, req.args)
213
+ self.dbdict[hsh] = r
214
+ self.dbdict.commit()
215
+
216
+ return res
217
+
218
+ return fn
219
+
220
+ def get_cache_hook(self):
221
+ return CacheHook(self)
LaViDa/eval/lmms_eval/api/registry.py ADDED
@@ -0,0 +1,185 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from typing import Callable, Dict, Union
2
+
3
+ import evaluate as hf_evaluate
4
+ from loguru import logger as eval_logger
5
+
6
+ from lmms_eval.api.model import lmms
7
+
8
+ MODEL_REGISTRY = {}
9
+
10
+
11
+ def register_model(*names):
12
+ # either pass a list or a single alias.
13
+ # function receives them as a tuple of strings
14
+
15
+ def decorate(cls):
16
+ for name in names:
17
+ assert issubclass(cls, lmms), f"Model '{name}' ({cls.__name__}) must extend lmms class"
18
+
19
+ assert name not in MODEL_REGISTRY, f"Model named '{name}' conflicts with existing model! Please register with a non-conflicting alias instead."
20
+
21
+ MODEL_REGISTRY[name] = cls
22
+ return cls
23
+
24
+ return decorate
25
+
26
+
27
+ def get_model(model_name):
28
+ try:
29
+ return MODEL_REGISTRY[model_name]
30
+ except KeyError:
31
+ raise ValueError(f"Attempted to load model '{model_name}', but no model for this name found! Supported model names: {', '.join(MODEL_REGISTRY.keys())}")
32
+
33
+
34
+ TASK_REGISTRY = {} # Key: task name, Value: task ConfigurableTask class
35
+ GROUP_REGISTRY = {} # Key: group name, Value: list of task names or group names
36
+ TASK_INITIALIZED = False
37
+ ALL_TASKS = set() # Set of all task names and group names
38
+ func2task_index = {} # Key: task ConfigurableTask class, Value: task name
39
+ OUTPUT_TYPE_REGISTRY = {}
40
+ METRIC_REGISTRY = {}
41
+ METRIC_AGGREGATION_REGISTRY = {}
42
+ AGGREGATION_REGISTRY: Dict[str, Callable[[], Dict[str, Callable]]] = {}
43
+ HIGHER_IS_BETTER_REGISTRY = {}
44
+ FILTER_REGISTRY = {}
45
+
46
+
47
+ def register_task(name):
48
+ def decorate(fn):
49
+ assert name not in TASK_REGISTRY, f"task named '{name}' conflicts with existing registered task!"
50
+
51
+ TASK_REGISTRY[name] = fn
52
+ ALL_TASKS.add(name)
53
+ func2task_index[fn.__name__] = name
54
+ return fn
55
+
56
+ return decorate
57
+
58
+
59
+ def register_group(name):
60
+ def decorate(fn):
61
+ func_name = func2task_index[fn.__name__]
62
+ if name in GROUP_REGISTRY:
63
+ GROUP_REGISTRY[name].append(func_name)
64
+ else:
65
+ GROUP_REGISTRY[name] = [func_name]
66
+ ALL_TASKS.add(name)
67
+ return fn
68
+
69
+ return decorate
70
+
71
+
72
+ OUTPUT_TYPE_REGISTRY = {}
73
+ METRIC_REGISTRY = {}
74
+ METRIC_AGGREGATION_REGISTRY = {}
75
+ AGGREGATION_REGISTRY = {}
76
+ HIGHER_IS_BETTER_REGISTRY = {}
77
+
78
+ DEFAULT_METRIC_REGISTRY = {
79
+ "loglikelihood": [
80
+ "perplexity",
81
+ "acc",
82
+ ],
83
+ "multiple_choice": ["acc", "acc_norm"],
84
+ "generate_until": ["exact_match"],
85
+ "generate_until_multi_round": ["exact_match"],
86
+ }
87
+
88
+
89
+ def register_metric(**args):
90
+ # TODO: do we want to enforce a certain interface to registered metrics?
91
+ def decorate(fn):
92
+ assert "metric" in args
93
+ name = args["metric"]
94
+
95
+ for key, registry in [
96
+ ("metric", METRIC_REGISTRY),
97
+ ("higher_is_better", HIGHER_IS_BETTER_REGISTRY),
98
+ ("aggregation", METRIC_AGGREGATION_REGISTRY),
99
+ ]:
100
+ if key in args:
101
+ value = args[key]
102
+ assert value not in registry, f"{key} named '{value}' conflicts with existing registered {key}!"
103
+
104
+ if key == "metric":
105
+ registry[name] = fn
106
+ elif key == "aggregation":
107
+ registry[name] = AGGREGATION_REGISTRY[value]
108
+ else:
109
+ registry[name] = value
110
+
111
+ return fn
112
+
113
+ return decorate
114
+
115
+
116
+ def get_metric(name: str, hf_evaluate_metric=False) -> Callable:
117
+ if not hf_evaluate_metric:
118
+ if name in METRIC_REGISTRY:
119
+ return METRIC_REGISTRY[name]
120
+ else:
121
+ eval_logger.warning(f"Could not find registered metric '{name}' in lm-eval, searching in HF Evaluate library...")
122
+
123
+ try:
124
+ metric_object = hf_evaluate.load(name)
125
+ return metric_object.compute
126
+ except Exception:
127
+ eval_logger.error(
128
+ f"{name} not found in the evaluate library! Please check https://huggingface.co/evaluate-metric",
129
+ )
130
+
131
+
132
+ def register_aggregation(name):
133
+ def decorate(fn):
134
+ assert name not in AGGREGATION_REGISTRY, f"aggregation named '{name}' conflicts with existing registered aggregation!"
135
+
136
+ AGGREGATION_REGISTRY[name] = fn
137
+ return fn
138
+
139
+ return decorate
140
+
141
+
142
+ def get_aggregation(name):
143
+ try:
144
+ return AGGREGATION_REGISTRY[name]
145
+ except KeyError:
146
+ eval_logger.warning(
147
+ "{} not a registered aggregation metric!".format(name),
148
+ )
149
+
150
+
151
+ def get_metric_aggregation(name):
152
+ try:
153
+ return METRIC_AGGREGATION_REGISTRY[name]
154
+ except KeyError:
155
+ eval_logger.warning(
156
+ "{} metric is not assigned a default aggregation!".format(name),
157
+ )
158
+
159
+
160
+ def is_higher_better(metric_name):
161
+ try:
162
+ return HIGHER_IS_BETTER_REGISTRY[metric_name]
163
+ except KeyError:
164
+ eval_logger.warning(f"higher_is_better not specified for metric '{metric_name}'!")
165
+
166
+
167
+ def register_filter(name):
168
+ def decorate(cls):
169
+ if name in FILTER_REGISTRY:
170
+ eval_logger.info(f"Registering filter `{name}` that is already in Registry {FILTER_REGISTRY}")
171
+ FILTER_REGISTRY[name] = cls
172
+ return cls
173
+
174
+ return decorate
175
+
176
+
177
+ def get_filter(filter_name: Union[str, Callable]) -> Callable:
178
+ try:
179
+ return FILTER_REGISTRY[filter_name]
180
+ except KeyError as e:
181
+ if callable(filter_name):
182
+ return filter_name
183
+ else:
184
+ eval_logger.warning(f"filter `{filter_name}` is not registered!")
185
+ raise e
LaViDa/eval/lmms_eval/api/samplers.py ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ class ContextSampler:
2
+ def __init__(self, docs, task, fewshot_indices=None, rnd=None) -> None:
3
+ self.rnd = rnd
4
+ assert self.rnd, "must pass rnd to FewShotSampler!"
5
+
6
+ self.task = task
7
+ self.config = task._config
8
+
9
+ self.target_delimiter = self.config.target_delimiter
10
+ self.fewshot_delimiter = self.config.fewshot_delimiter
11
+
12
+ self.doc_to_text = self.task.doc_to_text
13
+ self.doc_to_target = self.task.doc_to_target
14
+ self.doc_to_choice = self.task.doc_to_choice
15
+
16
+ self.docs = docs # HF dataset split, provided by task._fewshot_docs()
17
+ if fewshot_indices: # subset few-shot docs from
18
+ self.docs = self.docs.select(fewshot_indices)
19
+
20
+ def get_context(self, doc, num_fewshot):
21
+ # draw an extra fewshot sample if using same split as evaluating on
22
+ n_samples = num_fewshot + 1 if self.config.fewshot_split == self.config.test_split else num_fewshot
23
+
24
+ # draw `n_samples` docs from fewshot_docs
25
+ fewshotex = self.sample(n_samples)
26
+
27
+ # get rid of the doc that's the one we're evaluating, if it's in the fewshot
28
+ # TODO: should we just stop people from using fewshot from same split as evaluating?
29
+ selected_docs = [x for x in fewshotex if x != doc][:num_fewshot]
30
+
31
+ labeled_examples = (
32
+ self.fewshot_delimiter.join(
33
+ [
34
+ # TODO: is separating doc_to_text and doc_to_target by one space always desired?
35
+ (self.doc_to_text(doc) if (self.config.doc_to_choice is None or type(self.doc_to_text(doc)) is str) else self.doc_to_choice(doc)[self.doc_to_text(doc)])
36
+ + self.target_delimiter
37
+ + (
38
+ str(self.doc_to_target(doc)[0])
39
+ if type(self.doc_to_target(doc)) is list
40
+ else self.doc_to_target(doc)
41
+ if (self.config.doc_to_choice is None or type(self.doc_to_target(doc)) is str)
42
+ else str(self.doc_to_choice(doc)[self.doc_to_target(doc)])
43
+ )
44
+ for doc in selected_docs
45
+ ]
46
+ )
47
+ + self.fewshot_delimiter
48
+ )
49
+
50
+ return labeled_examples
51
+
52
+ def sample(self, n):
53
+ """
54
+ Draw `n` samples from our fewshot docs. This method should be overridden by subclasses.
55
+ """
56
+
57
+ return self.rnd.sample(self.docs, n)
58
+
59
+
60
+ class FirstNSampler(ContextSampler):
61
+ def sample(self, n) -> None:
62
+ """
63
+ Draw the first `n` samples in order from the specified split.
64
+ Used for tasks with "canonical" ordered fewshot examples, such as MMLU and CMMLU.
65
+ """
66
+ assert n <= len(self.docs), f"Error: number of fewshot samples requested exceeds the {len(self.docs)} that are available."
67
+ return self.docs[:n]
68
+
69
+
70
+ class BalancedSampler(ContextSampler):
71
+ def sample(self, n) -> None:
72
+ """
73
+ TODO: this should return approximately class-balanced samples from our fewshot examples.
74
+ TODO: what order should they be in? maybe random?
75
+ """
76
+
77
+ pass
78
+
79
+
80
+ class ManualSampler(ContextSampler):
81
+ def sample(self, n) -> None:
82
+ """ """
83
+ pass
84
+
85
+
86
+ SAMPLER_REGISTRY = {
87
+ "default": ContextSampler,
88
+ "first_n": FirstNSampler,
89
+ }
90
+
91
+
92
+ def get_sampler(name):
93
+ try:
94
+ return SAMPLER_REGISTRY[name]
95
+ except KeyError:
96
+ raise ValueError(f"Attempted to use contextsampler '{name}', but no sampling strategy for this name found! Supported model names: {', '.join(SAMPLER_REGISTRY.keys())}")
LaViDa/eval/lmms_eval/api/task.py ADDED
@@ -0,0 +1,1630 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import abc
2
+ import ast
3
+ import copy
4
+ import inspect
5
+ import itertools
6
+ import json
7
+ import os
8
+ import random
9
+ import re
10
+ import shutil
11
+ import subprocess
12
+ from collections.abc import Callable
13
+ from dataclasses import asdict, dataclass, field
14
+ from functools import partial
15
+ from glob import glob
16
+ from typing import (
17
+ Any,
18
+ Dict,
19
+ Iterable,
20
+ Iterator,
21
+ List,
22
+ Literal,
23
+ Mapping,
24
+ Optional,
25
+ Tuple,
26
+ Union,
27
+ )
28
+
29
+ import datasets
30
+ import numpy as np
31
+ from accelerate import Accelerator
32
+ from datasets import Audio, DownloadConfig, Image, Sequence
33
+ from huggingface_hub import snapshot_download
34
+ from loguru import logger as eval_logger
35
+ from PIL import ImageFile
36
+ from tenacity import retry, stop_after_attempt, stop_after_delay, wait_fixed
37
+ from tqdm import tqdm
38
+
39
+ from lmms_eval import utils
40
+ from lmms_eval.api import samplers
41
+ from lmms_eval.api.instance import Instance
42
+ from lmms_eval.api.registry import (
43
+ AGGREGATION_REGISTRY,
44
+ DEFAULT_METRIC_REGISTRY,
45
+ METRIC_REGISTRY,
46
+ OUTPUT_TYPE_REGISTRY,
47
+ get_aggregation,
48
+ get_metric,
49
+ get_metric_aggregation,
50
+ is_higher_better,
51
+ )
52
+ from lmms_eval.caching.cache import load_from_cache, save_to_cache
53
+ from lmms_eval.filters import build_filter_ensemble
54
+
55
+ # HuggingfaceM4/NoCaps contains truncated image in test split
56
+ # Include this inside code block to avoid error
57
+ ImageFile.LOAD_TRUNCATED_IMAGES = True
58
+
59
+ ALL_OUTPUT_TYPES = [
60
+ "loglikelihood",
61
+ "multiple_choice",
62
+ "generate_until",
63
+ "generate_until_multi_round",
64
+ ]
65
+
66
+
67
+ @dataclass
68
+ class TaskConfig(dict):
69
+ # task naming/registry
70
+ task: str = None
71
+ task_alias: str = None
72
+ tag: str = None
73
+ group: Union[str, list] = None
74
+ group_alias: Union[str, list] = None
75
+ # HF dataset options.
76
+ # which dataset to use,
77
+ # and what splits for what purpose
78
+ dataset_path: str = None
79
+ dataset_name: str = None
80
+ dataset_kwargs: dict = None
81
+ training_split: str = None
82
+ validation_split: str = None
83
+ test_split: str = None
84
+ fewshot_split: str = None # TODO: assert that this not None if num_fewshot > 0. (?) assert if this is same split as one evaling (?)
85
+ full_docs: bool = False
86
+ # formatting / prompting options.
87
+ # see docs/advanced_task_guide.md for more info
88
+ process_results_use_image: bool = False
89
+ process_docs: Callable = None
90
+ doc_to_visual: Union[Callable, str] = None
91
+ doc_to_text: Union[Callable, str] = None
92
+ doc_to_target: Union[Callable, str] = None
93
+ doc_to_choice: Union[Callable, str, dict, list] = None
94
+ process_results: Union[Callable, str] = None
95
+ use_prompt: str = None
96
+ description: str = ""
97
+ target_delimiter: str = " "
98
+ fewshot_delimiter: str = "\n\n"
99
+ fewshot_config: dict = None
100
+ # runtime configuration options
101
+ num_fewshot: int = None
102
+ # scoring options
103
+ metric_list: list = None
104
+ output_type: str = "generate_until"
105
+ generation_kwargs: dict = None
106
+ repeats: int = 1
107
+ filter_list: Union[str, list] = None
108
+ should_decontaminate: bool = False
109
+ doc_to_decontamination_query: str = None
110
+
111
+ metadata: Union[str, list] = None # by default, not used in the code. allows for users to pass arbitrary info to tasks
112
+
113
+ lmms_eval_specific_kwargs: dict = None
114
+ model_specific_generation_kwargs: dict = None
115
+ model_specific_target_kwargs: dict = None
116
+
117
+ def __post_init__(self) -> None:
118
+ if self.dataset_path and os.path.exists(os.path.dirname(self.dataset_path)):
119
+ import inspect
120
+ from importlib import import_module
121
+
122
+ # self.dataset_path = inspect.getfile(import_module(self.dataset_path))
123
+
124
+ if self.group is not None:
125
+ eval_logger.warning(
126
+ "A task YAML file was found to contain a `group` key. Groups which provide aggregate scores over several subtasks now require a separate config file--if not aggregating, you may want to use the `tag` config option instead within your config. Setting `group` within a TaskConfig will be deprecated in v0.4.4. Please see https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md for more information."
127
+ )
128
+
129
+ if self.tag is None:
130
+ self.tag = self.group
131
+ else:
132
+ raise ValueError("Got both a `group` and `tag` entry within a TaskConfig. Please use one or the other--`group` values will be deprecated in v0.4.4.")
133
+
134
+ if self.generation_kwargs is not None:
135
+ if "generate_until" not in self.output_type:
136
+ eval_logger.warning(f"[{self.task}] passed `generation_kwargs`, but not using `output_type: generate_until`!")
137
+ assert "generate_until" not in self.output_type
138
+
139
+ if "temperature" in self.generation_kwargs:
140
+ self.generation_kwargs["temperature"] = float(self.generation_kwargs["temperature"])
141
+
142
+ if "until" not in self.generation_kwargs:
143
+ self.generation_kwargs["until"] = [self.fewshot_delimiter]
144
+ else:
145
+ if "generate_until" in self.output_type:
146
+ # ensure that we greedily generate in absence of explicit arguments otherwise
147
+ self.generation_kwargs = {
148
+ "until": None if self.fewshot_delimiter is None else [self.fewshot_delimiter],
149
+ "do_sample": False,
150
+ }
151
+
152
+ # TODO: how to make TaskConfigs be de- and re-serializable, even when using the !function constructor?
153
+
154
+ def __getitem__(self, item):
155
+ return getattr(self, item)
156
+
157
+ def __setitem__(self, item, value):
158
+ return setattr(self, item, value)
159
+
160
+ def to_dict(self):
161
+ """dumps the current config as a dictionary object, as a printable format.
162
+ null fields will not be printed.
163
+ Used for dumping results alongside full task configuration
164
+
165
+ :return: dict
166
+ A printable dictionary version of the TaskConfig object.
167
+
168
+ # TODO: should any default value in the TaskConfig not be printed?
169
+ """
170
+ cfg_dict = asdict(self)
171
+ # remove values that are `None`
172
+ for k, v in list(cfg_dict.items()):
173
+ if v is None:
174
+ cfg_dict.pop(k)
175
+ elif isinstance(v, Callable):
176
+ # TODO: this should handle Promptsource template objects as a separate case?
177
+ cfg_dict[k] = str(v)
178
+ return cfg_dict
179
+
180
+
181
+ class Task(abc.ABC):
182
+ """A task represents an entire benchmark including its dataset, problems,
183
+ answers, and evaluation methods. See BoolQ for a simple example implementation
184
+
185
+ A `doc` can be any python object which represents one instance of evaluation.
186
+ This is usually a dictionary e.g.
187
+ {"question": ..., "answer": ...} or
188
+ {"question": ..., question, answer)
189
+ """
190
+
191
+ VERSION = None
192
+
193
+ # The name of the `Task` benchmark as denoted in the HuggingFace datasets Hub
194
+ # or a path to a custom `datasets` loading script.
195
+ DATASET_PATH: str = None
196
+
197
+ # The name of a subset within `DATASET_PATH`.
198
+ DATASET_NAME: str = None
199
+
200
+ OUTPUT_TYPE: str = None
201
+
202
+ def __init__(
203
+ self,
204
+ data_dir=None,
205
+ cache_dir=None,
206
+ download_mode=None,
207
+ config=None,
208
+ ) -> None:
209
+ """
210
+ :param data_dir: str
211
+ Stores the path to a local folder containing the `Task`'s data files.
212
+ Use this to specify the path to manually downloaded data (usually when
213
+ the dataset is not publicly accessible).
214
+ :param cache_dir: str
215
+ The directory to read/write the `Task` dataset. This follows the
216
+ HuggingFace `datasets` API with the default cache directory located at:
217
+ `~/.cache/huggingface/datasets`
218
+ NOTE: You can change the cache location globally for a given process
219
+ to another directory:
220
+ `export HF_DATASETS_CACHE="/path/to/another/directory"`
221
+ :param download_mode: datasets.DownloadMode
222
+ How to treat pre-existing `Task` downloads and data.
223
+ - `datasets.DownloadMode.REUSE_DATASET_IF_EXISTS`
224
+ Reuse download and reuse dataset.
225
+ - `datasets.DownloadMode.REUSE_CACHE_IF_EXISTS`
226
+ Reuse download with fresh dataset.
227
+ - `datasets.DownloadMode.FORCE_REDOWNLOAD`
228
+ Fresh download and fresh dataset.
229
+ """
230
+ self.download(data_dir, cache_dir, download_mode)
231
+ self._training_docs = None
232
+ self._fewshot_docs = None
233
+ self._instances = None
234
+
235
+ self._config = TaskConfig({**config}) if config else TaskConfig()
236
+
237
+ self._filters = [build_filter_ensemble("none", [["take_first", None]])]
238
+
239
+ def download(self, data_dir=None, cache_dir=None, download_mode=None) -> None:
240
+ """Downloads and returns the task dataset.
241
+ Override this method to download the dataset from a custom API.
242
+
243
+ :param data_dir: str
244
+ Stores the path to a local folder containing the `Task`'s data files.
245
+ Use this to specify the path to manually downloaded data (usually when
246
+ the dataset is not publicly accessible).
247
+ :param cache_dir: str
248
+ The directory to read/write the `Task` dataset. This follows the
249
+ HuggingFace `datasets` API with the default cache directory located at:
250
+ `~/.cache/huggingface/datasets`
251
+ NOTE: You can change the cache location globally for a given process
252
+ by setting the shell environment variable, `HF_DATASETS_CACHE`,
253
+ to another directory:
254
+ `export HF_DATASETS_CACHE="/path/to/another/directory"`
255
+ :param download_mode: datasets.DownloadMode
256
+ How to treat pre-existing `Task` downloads and data.
257
+ - `datasets.DownloadMode.REUSE_DATASET_IF_EXISTS`
258
+ Reuse download and reuse dataset.
259
+ - `datasets.DownloadMode.REUSE_CACHE_IF_EXISTS`
260
+ Reuse download with fresh dataset.
261
+ - `datasets.DownloadMode.FORCE_REDOWNLOAD`
262
+ Fresh download and fresh dataset.
263
+ """
264
+ self.dataset = datasets.load_dataset(
265
+ path=self.DATASET_PATH,
266
+ name=self.DATASET_NAME,
267
+ data_dir=data_dir,
268
+ cache_dir=cache_dir,
269
+ download_mode=download_mode,
270
+ )
271
+ self.dataset_no_image = datasets.load_dataset(
272
+ path=self.DATASET_PATH,
273
+ name=self.DATASET_NAME,
274
+ data_dir=data_dir,
275
+ cache_dir=cache_dir,
276
+ download_mode=download_mode,
277
+ )
278
+ for doc_name in self.dataset_no_image:
279
+ remove_cols = []
280
+ features = self.dataset_no_image[doc_name].features
281
+ # If it is an Image instance or a Sequence of Image instance. Remove it
282
+ for feature in features:
283
+ if isinstance(features[feature], Image):
284
+ remove_cols.append(feature)
285
+ elif isinstance(features[feature], Sequence) and isinstance(features[feature].feature, Image):
286
+ remove_cols.append(feature)
287
+ for remove_col in remove_cols:
288
+ self.dataset_no_image[doc_name] = self.dataset_no_image[doc_name].remove_columns(remove_col)
289
+
290
+ @property
291
+ def config(self):
292
+ """Returns the TaskConfig associated with this class."""
293
+ return self._config
294
+
295
+ @abc.abstractmethod
296
+ def has_training_docs(self):
297
+ """Whether the task has a training set"""
298
+ pass
299
+
300
+ @abc.abstractmethod
301
+ def has_validation_docs(self):
302
+ """Whether the task has a validation set"""
303
+ pass
304
+
305
+ @abc.abstractmethod
306
+ def has_test_docs(self):
307
+ """Whether the task has a test set"""
308
+ pass
309
+
310
+ def training_docs(self):
311
+ """
312
+ :return: Iterable[obj]
313
+ A iterable of any object, that doc_to_text can handle
314
+ """
315
+ return []
316
+
317
+ def validation_docs(self):
318
+ """
319
+ :return: Iterable[obj]
320
+ A iterable of any object, that doc_to_text can handle
321
+ """
322
+ return []
323
+
324
+ def test_docs(self):
325
+ """
326
+ :return: Iterable[obj]
327
+ A iterable of any object, that doc_to_text can handle
328
+ """
329
+ return []
330
+
331
+ def fewshot_docs(self):
332
+ """
333
+ :return: Iterable[obj]
334
+ A iterable of any object, that doc_to_text can handle
335
+ """
336
+ if self.has_training_docs():
337
+ return self.training_docs()
338
+ elif self.has_validation_docs():
339
+ return self.validation_docs()
340
+ else:
341
+ if self.config.num_fewshot is not None:
342
+ eval_logger.warning("has_training_docs and has_validation_docs are False" ", using test_docs as fewshot_docs but this is not recommended.")
343
+ return self.test_docs()
344
+
345
+ def _process_doc(self, doc):
346
+ """
347
+ Override this to process (detokenize, strip, replace, etc.) individual
348
+ documents. This can be used in a map over documents of a data split.
349
+ E.g. `map(self._process_doc, self.dataset["validation"])`
350
+
351
+ :return: dict
352
+ The processed version of the specified `doc`.
353
+ """
354
+ return doc
355
+
356
+ @property
357
+ def instances(self):
358
+ """After calling `task.build_all_requests()`, tasks
359
+ maintain a list of the dataset instances which will be evaluated.
360
+ """
361
+ return self._instances
362
+
363
+ def fewshot_examples(self, k, rnd):
364
+ if self._training_docs is None:
365
+ self._training_docs = list(self.training_docs())
366
+
367
+ return rnd.sample(self._training_docs, k)
368
+
369
+ def doc_to_decontamination_query(self, doc) -> None:
370
+ print("Override doc_to_decontamination_query with document specific decontamination query.")
371
+ assert False
372
+
373
+ @abc.abstractmethod
374
+ def doc_to_text(self, doc):
375
+ pass
376
+
377
+ @abc.abstractmethod
378
+ def doc_to_target(self, doc):
379
+ pass
380
+
381
+ # @profile
382
+ def build_all_requests(
383
+ self,
384
+ *,
385
+ limit: Union[int, None] = None,
386
+ rank: int = 0,
387
+ world_size: int = 1,
388
+ cache_requests: bool = False,
389
+ rewrite_requests_cache: bool = False,
390
+ system_instruction: Optional[str] = None,
391
+ apply_chat_template: bool = False,
392
+ fewshot_as_multiturn: bool = False,
393
+ chat_template: Optional[Callable] = None,
394
+ tokenizer_name: str = "",
395
+ ) -> None:
396
+ """Build a set of Instances for a task, and store them in task.instances"""
397
+ if self.has_test_docs():
398
+ docs = self.test_docs()
399
+ split = self.config.test_split
400
+ elif self.has_validation_docs():
401
+ docs = self.validation_docs()
402
+ split = self.config.validation_split
403
+ else:
404
+ assert False, f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!"
405
+
406
+ # used with caching
407
+ og_limit = limit
408
+
409
+ cache_key = f"requests-{self._config.task}-{self.config.num_fewshot}shot-rank{rank}-world_size{world_size}"
410
+ cache_key += "-chat_template" if apply_chat_template else ""
411
+ cache_key += "-fewshot_as_multiturn" if fewshot_as_multiturn else ""
412
+ cache_key += f"-system_prompt_hash{utils.hash_string(system_instruction)}" if system_instruction is not None else ""
413
+ cache_key += f"-tokenizer{tokenizer_name}"
414
+
415
+ cached_instances = load_from_cache(file_name=cache_key)
416
+
417
+ if cache_requests and cached_instances and not rewrite_requests_cache:
418
+ cached_instances = cached_instances[:limit]
419
+
420
+ flattened_instances = [instance for instance_group in cached_instances for instance in instance_group]
421
+
422
+ self._instances = flattened_instances
423
+ return
424
+
425
+ eval_logger.info(f"Building contexts for {self.config.task} on rank {rank}...")
426
+
427
+ instances = []
428
+
429
+ # process all documents when caching is specified for simplicity
430
+ if cache_requests and (not cached_instances or rewrite_requests_cache) and limit is not None:
431
+ limit = None
432
+
433
+ doc_id_docs = utils.create_iterator(enumerate(self.eval_docs_no_media), rank=rank, limit=int(limit) if limit else None, world_size=world_size)
434
+ doc_iterator_for_counting = itertools.islice(range(len(self.test_docs())), rank, limit, world_size) if self.has_test_docs() else itertools.islice(range(len(self.validation_docs())), rank, limit, world_size)
435
+
436
+ num_docs = sum(1 for _ in doc_iterator_for_counting)
437
+
438
+ for doc_id, doc in tqdm(
439
+ doc_id_docs,
440
+ total=num_docs,
441
+ ):
442
+ # sample fewshot context #TODO: need to offset doc_id by rank now!
443
+ fewshot_ctx = self.fewshot_context(
444
+ doc,
445
+ 0 if self.config.num_fewshot is None else self.config.num_fewshot,
446
+ system_instruction,
447
+ apply_chat_template,
448
+ fewshot_as_multiturn,
449
+ chat_template,
450
+ )
451
+
452
+ # TODO: we should override self.config.repeats if doing greedy gen so users don't waste time+compute
453
+ per_task_metadata = {"task": self.config["task"], "doc_id": doc_id, "repeats": self.config.repeats, "split": split}
454
+ if self.config.metadata and type(self.config.metadata) == dict: # TODO: temporary fix for metadata loading, ignore the list of dict type.
455
+ per_task_metadata.update(self.config.metadata)
456
+
457
+ inst = self.construct_requests(doc_id=doc_id, ctx=fewshot_ctx, metadata=per_task_metadata)
458
+
459
+ if not isinstance(inst, list):
460
+ inst = [inst]
461
+
462
+ instances.append(inst)
463
+
464
+ # now flatten, this is to allow slicing to work with pickles
465
+
466
+ sliced_instances = instances[:og_limit]
467
+
468
+ flattened_instances = [instance for instance_group in sliced_instances for instance in instance_group]
469
+
470
+ self._instances = flattened_instances
471
+
472
+ if len(self._instances) == 0:
473
+ raise ValueError("task.build_requests() did not find any docs!")
474
+
475
+ if cache_requests and (not cached_instances or rewrite_requests_cache):
476
+ save_to_cache(file_name=cache_key, obj=instances)
477
+
478
+ # FIXME: Bo - We need to check if the doc_to_visual if it's exists and restore it. If we use cache, the doc_to_visual will be None since it's not serializable
479
+ for instance in self._instances:
480
+ if instance.arguments[2] is None:
481
+ arguments = (instance.arguments[0], instance.arguments[1], self.doc_to_visual, *instance.arguments[3:])
482
+ else:
483
+ arguments = instance.arguments
484
+
485
+ instance.arguments = arguments
486
+
487
+ @abc.abstractmethod
488
+ def construct_requests(self, doc_id, ctx, **kwargs):
489
+ """Uses RequestFactory to construct Requests and returns an iterable of
490
+ Requests which will be sent to the LMM.
491
+
492
+ :param doc_id: int
493
+ The index of a document within `self.test_docs()` or `self.validation_docs()`,
494
+ whichever is the main split used.
495
+ :param ctx: str
496
+ The context string, generated by fewshot_context. This includes the natural
497
+ language description, as well as the few shot examples, and the question
498
+ part of the document for `doc`.
499
+ :param repeats: int
500
+ TODO: update this docstring
501
+ The number of times each instance in a dataset is inferred on. Defaults to 1,
502
+ can be increased for techniques like majority voting.
503
+ """
504
+ pass
505
+
506
+ @abc.abstractmethod
507
+ def process_results(self, doc, results):
508
+ """Take a single document and the LMM results and evaluates, returning a
509
+ dict where keys are the names of submetrics and values are the values of
510
+ the metric for that one document
511
+
512
+ :param doc:
513
+ The document as returned from training_docs, validation_docs, or test_docs.
514
+ :param results:
515
+ The results of the requests created in construct_requests.
516
+ """
517
+ pass
518
+
519
+ @abc.abstractmethod
520
+ def aggregation(self):
521
+ """
522
+ :returns: {str: [metric_score] -> float}
523
+ A dictionary where keys are the names of submetrics and values are
524
+ functions that aggregate a list of metric scores
525
+ """
526
+ pass
527
+
528
+ @abc.abstractmethod
529
+ def higher_is_better(self):
530
+ """
531
+ :returns: {str: bool}
532
+ A dictionary where keys are the names of submetrics and values are
533
+ whether a higher value of the submetric is better
534
+ """
535
+ pass
536
+
537
+ @classmethod
538
+ def count_bytes(cls, doc):
539
+ """Used for byte-level perplexity metrics in rolling loglikelihood"""
540
+ return len(doc.encode("utf-8"))
541
+
542
+ @utils.positional_deprecated
543
+ def fewshot_context(
544
+ self,
545
+ doc_id,
546
+ num_fewshot,
547
+ split,
548
+ rnd=random.Random(1234),
549
+ description=None,
550
+ ):
551
+ """Returns a fewshot context string that is made up of a prepended description
552
+ (if provided), the `num_fewshot` number of examples, and an appended prompt example.
553
+
554
+ :param doc_id: int
555
+ The document id as returned from training_docs, validation_docs, or test_docs.
556
+ :param num_fewshot: int
557
+ The number of fewshot examples to provide in the returned context string.
558
+ :param split: str
559
+ The split of the document to retrieve from the dataset
560
+ :param rnd: random.Random
561
+ The pseudo-random number generator used to randomly sample examples.
562
+ WARNING: This is currently a required arg although it's optionalized with a default `None`.
563
+ :param description: str
564
+ The task's description that will be prepended to the fewshot examples.
565
+ :returns: str
566
+ The fewshot context.
567
+ """
568
+ assert rnd is not None, "A `random.Random` generator argument must be provided to `rnd`"
569
+
570
+ description = description if description else ""
571
+ doc = self.dataset_no_image[split][doc_id]
572
+
573
+ if num_fewshot == 0:
574
+ labeled_examples = ""
575
+ else:
576
+ # for sets with no training docs, draw from other set *but ensure no overlap with current doc*
577
+ if self.has_training_docs():
578
+ fewshotex = self.fewshot_examples(k=num_fewshot, rnd=rnd)
579
+ else:
580
+ if self._fewshot_docs is None:
581
+ self._fewshot_docs = list(self.validation_docs() if self.has_validation_docs() else self.test_docs())
582
+
583
+ fewshotex = rnd.sample(self._fewshot_docs, num_fewshot + 1)
584
+
585
+ # get rid of the doc that's the one we're evaluating, if it's in the fewshot
586
+ fewshotex = [x for x in fewshotex if x != doc][:num_fewshot]
587
+
588
+ labeled_examples = "\n\n".join([self.doc_to_text(doc) + self.doc_to_target(doc) for doc in fewshotex]) + "\n\n"
589
+
590
+ example = self.doc_to_text(doc)
591
+ return description + labeled_examples + example
592
+
593
+ def apply_filters(self) -> Optional[List[Instance]]:
594
+ """Iterates over FilterEnsembles and applies them to instances"""
595
+ if hasattr(self, "_filters"):
596
+ for f in self._filters:
597
+ f.apply(self._instances, None)
598
+ else:
599
+ eval_logger.warning("No filter defined, passing through instances")
600
+ return self._instances
601
+
602
+ def dump_config(self) -> dict:
603
+ """Returns a dictionary representing the task's config.
604
+
605
+ :returns: str
606
+ The fewshot context.
607
+ """
608
+ # TODO: this should only return the overrides applied to a non-YAML task's configuration.
609
+ # (num_fewshot)
610
+ return self.config.to_dict()
611
+
612
+ def set_config(self, key: str, value: Any, update: bool = False) -> None:
613
+ """Set or update the configuration for a given key."""
614
+ if key is None:
615
+ raise ValueError("Key must be provided.")
616
+
617
+ if update:
618
+ current_value = getattr(self._config, key, {})
619
+ if not isinstance(current_value, dict):
620
+ raise TypeError(f"Expected a dict for key '{key}', got {type(current_value).__name__} instead.")
621
+ current_value.update(value)
622
+ else:
623
+ setattr(self._config, key, value)
624
+
625
+ def override_metric(self, metric_name: str) -> None:
626
+ """
627
+ Override the default metrics used for evaluation with custom metrics.
628
+
629
+ Parameters:
630
+ - metric_name (str): The name of the custom metric to override. Should be registered in api.metrics.
631
+ """
632
+ (
633
+ self._metric_fn_list,
634
+ self._aggregation_list,
635
+ self._metric_fn_kwargs,
636
+ self._higher_is_better,
637
+ ) = ({}, {}, {}, {})
638
+ self._metric_fn_list[metric_name] = get_metric(metric_name)
639
+ self._aggregation_list[metric_name] = get_metric_aggregation(metric_name)
640
+ self._higher_is_better[metric_name] = is_higher_better(metric_name)
641
+ self._metric_fn_kwargs[metric_name] = {}
642
+ if not isinstance(self, ConfigurableTask):
643
+ self.process_results = lambda x, y: {metric_name: get_metric(metric_name)}
644
+ self.aggregation = lambda: {metric_name: get_metric_aggregation(metric_name)}
645
+ setattr(self._config, "metric_list", [{"metric": metric_name}])
646
+ setattr(self._config, "process_results", None)
647
+
648
+ def set_fewshot_seed(self, seed: Optional[int] = None) -> None:
649
+ self.fewshot_rnd = random.Random(seed)
650
+ if hasattr(self, "sampler"):
651
+ self.sampler.rnd = self.fewshot_rnd
652
+
653
+ @property
654
+ def eval_docs(self) -> Union[datasets.Dataset, List[dict]]:
655
+ if self.has_test_docs():
656
+ return self.test_docs()
657
+ elif self.has_validation_docs():
658
+ return self.validation_docs()
659
+ else:
660
+ raise ValueError(f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!")
661
+
662
+ def doc_iterator(self, *, rank: int = 0, limit: Union[int, None] = None, world_size: int = 1) -> Iterator[Tuple[int, Any]]:
663
+ limit = int(limit) if limit else None
664
+ doc_iterator = utils.create_iterator(
665
+ enumerate(self.eval_docs),
666
+ rank=int(rank),
667
+ limit=limit,
668
+ world_size=int(world_size),
669
+ )
670
+ return doc_iterator
671
+
672
+
673
+ class ConfigurableTask(Task):
674
+ VERSION = "Yaml"
675
+ OUTPUT_TYPE = None
676
+ CONFIG = None
677
+
678
+ def __init__(
679
+ self,
680
+ data_dir=None,
681
+ cache_dir=None,
682
+ download_mode=None,
683
+ config: Optional[dict] = None,
684
+ model_name: Optional[str] = None,
685
+ ) -> None: # TODO no super() call here
686
+ # Get pre-configured attributes
687
+ self._config = self.CONFIG
688
+
689
+ # Use new configurations if there was no preconfiguration
690
+ if self.config is None:
691
+ self._config = TaskConfig(**config)
692
+ # Overwrite configs
693
+ else:
694
+ if config is not None:
695
+ self._config.__dict__.update(config)
696
+
697
+ if self.config is None:
698
+ raise ValueError("Must pass a config to ConfigurableTask, either in cls.CONFIG or `config` kwarg")
699
+
700
+ if isinstance(self.config.metadata, dict):
701
+ if "version" in self.config.metadata:
702
+ self.VERSION = self.config.metadata["version"]
703
+
704
+ self.model_name = model_name
705
+ self._prepare_model_specific_config()
706
+
707
+ if self.config.output_type is not None:
708
+ if self.config.output_type not in ALL_OUTPUT_TYPES:
709
+ raise ValueError(f"Got invalid output_type '{self.config.output_type}', must be in '{','.join(ALL_OUTPUT_TYPES)}'")
710
+ self.OUTPUT_TYPE = self.config.output_type
711
+
712
+ if self.config.dataset_path is not None:
713
+ self.DATASET_PATH = self.config.dataset_path
714
+
715
+ if self.config.dataset_name is not None:
716
+ self.DATASET_NAME = self.config.dataset_name
717
+
718
+ self._prepare_metric_and_aggregation()
719
+
720
+ self.download(self.config.dataset_kwargs)
721
+ self._training_docs = None
722
+ self._fewshot_docs = None
723
+
724
+ if self.config.filter_list is not None:
725
+ self._filters = []
726
+ for filter_config in self.config.filter_list:
727
+ for filter_pipeline in filter_config:
728
+ filter_name = filter_config["name"]
729
+ filter_functions = filter_config["filter"]
730
+ components = []
731
+ for function in filter_functions:
732
+ kwargs = {key: function[key] for key in function if key != "function"}
733
+ components.append([function["function"], kwargs])
734
+ filter_pipeline = build_filter_ensemble(filter_name, components)
735
+ self._filters.append(filter_pipeline)
736
+ else:
737
+ self._filters = [build_filter_ensemble("none", [["take_first", None]])]
738
+ if self.config.fewshot_config is not None:
739
+ self.sampler = samplers.get_sampler(self.config.fewshot_config.get("sampler", "default") if self.config.fewshot_config else "default")(list(self.fewshot_docs()), self, rnd=random.Random(1234))
740
+
741
+ if self.has_test_docs():
742
+ self.task_docs = self.test_docs()
743
+ elif self.has_validation_docs():
744
+ self.task_docs = self.validation_docs()
745
+ else:
746
+ assert False, f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!"
747
+
748
+ # Test One Doc
749
+ self.features = list(self.task_docs.features.keys())
750
+ self.multiple_input = 0
751
+ self.multiple_target = 0
752
+ test_doc = self.task_docs[0]
753
+ test_text = self.doc_to_text(test_doc)
754
+ test_target = self.doc_to_target(test_doc)
755
+
756
+ if self.config.doc_to_choice is not None:
757
+ test_choice = self.doc_to_choice(test_doc)
758
+ if type(test_choice) is not list:
759
+ eval_logger.error("doc_to_choice must return list")
760
+ else:
761
+ num_choice = len(test_choice)
762
+
763
+ if type(test_text) is int:
764
+ self.multiple_input = num_choice
765
+ else:
766
+ test_choice = None
767
+
768
+ if type(test_target) is list:
769
+ self.multiple_target = len(test_target)
770
+ else:
771
+ if (type(test_target) is int) and (test_choice is not None):
772
+ test_target = test_choice[test_target]
773
+ else:
774
+ test_target = str(test_target)
775
+
776
+ if test_choice is not None:
777
+ check_choices = test_choice
778
+ else:
779
+ check_choices = [test_target]
780
+ if self.config.doc_to_choice is not None:
781
+ for choice in check_choices:
782
+ choice_has_whitespace = True if choice[0].isspace() else False
783
+ delimiter_has_whitespace = True if self.config.target_delimiter.rstrip() != self.config.target_delimiter else False
784
+
785
+ if delimiter_has_whitespace and choice_has_whitespace:
786
+ eval_logger.warning(f'Both target_delimiter and target choice: "{choice}" have whitespace')
787
+ elif (not delimiter_has_whitespace) and (not choice_has_whitespace):
788
+ eval_logger.warning(f'Both target_delimiter "{self.config.target_delimiter}" and target choice: "{choice}" do not have whitespace, ignore if the language you are evaluating on does not require/use whitespace')
789
+
790
+ def _prepare_model_specific_config(self):
791
+ self.lmms_eval_specific_kwargs = self.config.lmms_eval_specific_kwargs
792
+ if self.lmms_eval_specific_kwargs is not None:
793
+ if self.model_name in self.lmms_eval_specific_kwargs:
794
+ self.lmms_eval_specific_kwargs = self.lmms_eval_specific_kwargs[self.model_name]
795
+ elif "default" in self.lmms_eval_specific_kwargs:
796
+ self.lmms_eval_specific_kwargs.update(self.lmms_eval_specific_kwargs.get("default", {}))
797
+ elif "dataset" in self.lmms_eval_specific_kwargs:
798
+ self.lmms_eval_specific_kwargs.update(self.lmms_eval_specific_kwargs.get("dataset", {}))
799
+
800
+ self.model_specific_target_kwargs = self.config.model_specific_target_kwargs
801
+ if self.model_specific_target_kwargs is not None:
802
+ if self.model_name in self.model_specific_target_kwargs:
803
+ self.model_specific_target_kwargs = self.model_specific_target_kwargs[self.model_name]
804
+ else:
805
+ self.model_specific_target_kwargs = self.model_specific_target_kwargs.get("default", None)
806
+ self.model_specific_generation_kwargs = self.config.model_specific_generation_kwargs
807
+ if self.model_specific_generation_kwargs is not None:
808
+ if self.model_name in self.model_specific_generation_kwargs:
809
+ self.model_specific_generation_kwargs = self.model_specific_generation_kwargs[self.model_name]
810
+ else:
811
+ self.model_specific_generation_kwargs = self.model_specific_generation_kwargs.get("default", {})
812
+
813
+ self.config.generation_kwargs.update(self.model_specific_generation_kwargs)
814
+
815
+ def _prepare_metric_and_aggregation(self):
816
+ self._metric_fn_list = {}
817
+ self._metric_fn_kwargs = {}
818
+ self._aggregation_list = {}
819
+ self._higher_is_better = {}
820
+
821
+ if self.config.metric_list is None:
822
+ # TODO: handle this in TaskConfig.__post_init__ ?
823
+ _metric_list = DEFAULT_METRIC_REGISTRY[self.config.output_type]
824
+
825
+ for metric_name in _metric_list:
826
+ self._metric_fn_list[metric_name] = METRIC_REGISTRY[metric_name]
827
+ self._metric_fn_kwargs[metric_name] = {}
828
+ self._aggregation_list[metric_name] = get_metric_aggregation(metric_name)
829
+ self._higher_is_better[metric_name] = is_higher_better(metric_name)
830
+ else:
831
+ for metric_config in self.config.metric_list:
832
+ assert "metric" in metric_config
833
+ metric_name = metric_config["metric"]
834
+ kwargs = {key: metric_config[key] for key in metric_config if key not in ["metric", "aggregation", "higher_is_better"]}
835
+
836
+ if self.config.process_results is not None:
837
+ self._metric_fn_list[metric_name] = None
838
+ self._metric_fn_kwargs[metric_name] = {}
839
+ elif callable(metric_name):
840
+ metric_fn = metric_name.__call__
841
+ metric_name = metric_name.__name__
842
+ self._metric_fn_list[metric_name] = metric_fn
843
+ self._metric_fn_kwargs[metric_name] = kwargs
844
+ else:
845
+ self._metric_fn_list[metric_name] = METRIC_REGISTRY[metric_name]
846
+ self._metric_fn_kwargs[metric_name] = kwargs
847
+
848
+ if "aggregation" in metric_config:
849
+ agg_name = metric_config["aggregation"]
850
+ if type(agg_name) == str:
851
+ self._aggregation_list[metric_name] = get_aggregation(agg_name)
852
+ elif callable(agg_name):
853
+ self._aggregation_list[metric_name] = metric_config["aggregation"]
854
+ else:
855
+ INV_AGG_REGISTRY = {v: k for k, v in AGGREGATION_REGISTRY.items()}
856
+ metric_agg = get_metric_aggregation(metric_name)
857
+ eval_logger.warning(f"[Task: {self._config.task}] metric {metric_name} is defined, but aggregation is not. " f"using default " f"aggregation={INV_AGG_REGISTRY[metric_agg]}")
858
+ self._aggregation_list[metric_name] = metric_agg
859
+
860
+ if "higher_is_better" in metric_config:
861
+ self._higher_is_better[metric_name] = metric_config["higher_is_better"]
862
+ else:
863
+ eval_logger.warning(f"[Task: {self._config.task}] metric {metric_name} is defined, but higher_is_better is not. " f"using default " f"higher_is_better={is_higher_better(metric_name)}")
864
+ self._higher_is_better[metric_name] = is_higher_better(metric_name)
865
+
866
+ @retry(stop=(stop_after_attempt(5) | stop_after_delay(60)), wait=wait_fixed(2))
867
+ def download(self, dataset_kwargs=None) -> None:
868
+ # If the dataset is a video dataset,
869
+ # Recursively search whether their is a zip and unzip it to the huggingface home
870
+ download_config = DownloadConfig()
871
+ download_config.max_retries = dataset_kwargs.get("max_retries", 10) if dataset_kwargs is not None else 10
872
+ download_config.num_proc = dataset_kwargs.get("num_proc", 8) if dataset_kwargs is not None else 8
873
+ download_config.local_files_only = dataset_kwargs.get("local_files_only", False) if dataset_kwargs is not None else False
874
+ if dataset_kwargs is not None:
875
+ if "From_YouTube" in dataset_kwargs:
876
+
877
+ def _download_from_youtube(path):
878
+ try:
879
+ for video in tqdm(self.all_dataset[split]):
880
+ video_id = video["videoID"]
881
+ target_path = os.path.join(path, f"{video_id}.mp4")
882
+ assert shutil.which("yt-dlp") is not None, "yt-dlp must be installed and available in the system's PATH"
883
+ command = f"yt-dlp -o {target_path} -f mp4 https://www.youtube.com/watch?v={video_id}"
884
+ subprocess.run(command, shell=True)
885
+ with open(os.path.join(cache_path, f"{task}_download_status.json"), "w") as f:
886
+ f.write(json.dumps({task: "downloaded"}))
887
+ except Exception as e:
888
+ eval_logger.error(f"Error while downloading {task} data: {e}")
889
+ with open(os.path.join(cache_path, f"{task}_download_status.json"), "w") as f:
890
+ f.write(json.dumps({task: "not downloaded"}))
891
+
892
+ hf_home = os.getenv("HF_HOME", "~/.cache/huggingface/")
893
+ accelerator = Accelerator()
894
+ if accelerator.is_main_process:
895
+ dataset_kwargs.pop("From_YouTube")
896
+ assert "load_from_disk" not in dataset_kwargs, "load_from_disk must not be True when From_YouTube is True"
897
+ self.all_dataset = datasets.load_dataset(
898
+ path=self.DATASET_PATH,
899
+ name=self.DATASET_NAME,
900
+ download_mode=datasets.DownloadMode.REUSE_DATASET_IF_EXISTS,
901
+ **dataset_kwargs if dataset_kwargs is not None else {},
902
+ )
903
+ dataset_kwargs["From_YouTube"] = True
904
+ cache_path = snapshot_download(repo_id=self.DATASET_PATH, repo_type="dataset") # download_parquet
905
+ split = vars(self.config)["test_split"]
906
+ task = vars(self.config)["task"]
907
+
908
+ video_path = os.path.join(hf_home, task)
909
+ if os.path.exists(os.path.join(cache_path, f"{task}_download_status.json")):
910
+ download_status = json.load(open(os.path.join(cache_path, f"{task}_download_status.json"), "r"))
911
+ if download_status[task] == "downloaded":
912
+ eval_logger.info(f"Data for {task} already download!")
913
+ else:
914
+ eval_logger.info(f"Start downloading YouTube data to {video_path}...")
915
+ _download_from_youtube(video_path)
916
+ else:
917
+ eval_logger.info(f"Start downloading YouTube data to {video_path}...")
918
+ _download_from_youtube(video_path)
919
+
920
+ accelerator.wait_for_everyone()
921
+ if "builder_script" in dataset_kwargs:
922
+ builder_script = dataset_kwargs["builder_script"]
923
+ self.DATASET_PATH = os.path.join(cache_path, builder_script)
924
+ dataset_kwargs.pop("builder_script")
925
+
926
+ downloaded_video_ids = [i.split(".mp4")[0] for i in os.listdir(os.path.expanduser(video_path)) if i.endswith(".mp4")]
927
+ # Filtered the existing dataset with the downloaded video ids
928
+ self.dataset = datasets.DatasetDict({split: self.all_dataset[split].filter(lambda x: x["videoID"] in downloaded_video_ids)})
929
+
930
+ self.dataset_no_image = self.dataset
931
+ dataset_kwargs.pop("From_YouTube")
932
+ return
933
+
934
+ if "video" in dataset_kwargs and dataset_kwargs["video"]:
935
+ hf_home = os.getenv("HF_HOME", "~/.cache/huggingface/")
936
+ hf_home = os.path.expanduser(hf_home)
937
+ cache_dir = dataset_kwargs["cache_dir"]
938
+ cache_dir = os.path.join(hf_home, cache_dir)
939
+ accelerator = Accelerator()
940
+ if accelerator.is_main_process:
941
+ force_download = dataset_kwargs.get("force_download", False)
942
+ force_unzip = dataset_kwargs.get("force_unzip", False)
943
+ revision = dataset_kwargs.get("revision", "main")
944
+ create_link = dataset_kwargs.get("create_link", False)
945
+ cache_path = snapshot_download(repo_id=self.DATASET_PATH, revision=revision, repo_type="dataset", force_download=force_download, etag_timeout=60)
946
+ zip_files = glob(os.path.join(cache_path, "**/*.zip"), recursive=True)
947
+ tar_files = glob(os.path.join(cache_path, "**/*.tar*"), recursive=True)
948
+
949
+ def unzip_video_data(zip_file):
950
+ import os
951
+ import zipfile
952
+
953
+ with zipfile.ZipFile(zip_file, "r") as zip_ref:
954
+ for file_info in zip_ref.infolist():
955
+ target_path = os.path.join(cache_dir, file_info.filename)
956
+ if not os.path.exists(target_path):
957
+ zip_ref.extract(file_info, cache_dir)
958
+ else:
959
+ eval_logger.info(f"Skipping existing file: {target_path}")
960
+
961
+ eval_logger.info(f"Extracted all files from {zip_file} to {cache_dir}")
962
+
963
+ def untar_video_data(tar_file):
964
+ import tarfile
965
+
966
+ with tarfile.open(tar_file, "r") as tar_ref:
967
+ tar_ref.extractall(cache_dir)
968
+ eval_logger.info(f"Extracted all files from {tar_file} to {cache_dir}")
969
+
970
+ def concat_tar_parts(tar_parts, output_tar):
971
+ with open(output_tar, "wb") as out_tar:
972
+ from tqdm import tqdm
973
+
974
+ for part in tqdm(sorted(tar_parts)):
975
+ with open(part, "rb") as part_file:
976
+ out_tar.write(part_file.read())
977
+ eval_logger.info(f"Concatenated parts {tar_parts} into {output_tar}")
978
+
979
+ # Unzip zip files if needed
980
+ if force_unzip or (not os.path.exists(cache_dir) and len(zip_files) > 0):
981
+ for zip_file in zip_files:
982
+ unzip_video_data(zip_file)
983
+
984
+ # Concatenate and extract tar files if needed
985
+ if force_unzip or (not os.path.exists(cache_dir) and len(tar_files) > 0):
986
+ tar_parts_dict = {}
987
+
988
+ # Group tar parts together
989
+ for tar_file in tar_files:
990
+ base_name = tar_file.split(".tar")[0]
991
+ if base_name not in tar_parts_dict:
992
+ tar_parts_dict[base_name] = []
993
+ tar_parts_dict[base_name].append(tar_file)
994
+
995
+ # Concatenate and untar split parts
996
+ for base_name, parts in tar_parts_dict.items():
997
+ eval_logger.info(f"Extracting following tar files: {parts}")
998
+ output_tar = base_name + ".tar"
999
+ if not os.path.exists(output_tar):
1000
+ eval_logger.info(f"Start concatenating tar files")
1001
+
1002
+ concat_tar_parts(parts, output_tar)
1003
+ eval_logger.info(f"Finish concatenating tar files")
1004
+
1005
+ if not os.path.exists(os.path.join(cache_dir, os.path.basename(base_name))):
1006
+ untar_video_data(output_tar)
1007
+
1008
+ # Link cache_path to cache_dir if needed.
1009
+ if create_link:
1010
+ if not os.path.exists(cache_dir) or os.path.islink(cache_dir):
1011
+ if os.path.islink(cache_dir):
1012
+ os.remove(cache_dir)
1013
+ eval_logger.info(f"Removed existing symbolic link: {cache_dir}")
1014
+ # Create a new symbolic link
1015
+ os.symlink(cache_path, cache_dir)
1016
+ eval_logger.info(f"Symbolic link created successfully: {cache_path} -> {cache_dir}")
1017
+
1018
+ accelerator.wait_for_everyone()
1019
+ dataset_kwargs.pop("cache_dir")
1020
+ dataset_kwargs.pop("video")
1021
+
1022
+ if "builder_script" in dataset_kwargs:
1023
+ builder_script = dataset_kwargs["builder_script"]
1024
+ self.DATASET_PATH = os.path.join(cache_path, builder_script)
1025
+ dataset_kwargs.pop("builder_script")
1026
+
1027
+ if "force_download" in dataset_kwargs:
1028
+ dataset_kwargs.pop("force_download")
1029
+
1030
+ if "force_unzip" in dataset_kwargs:
1031
+ dataset_kwargs.pop("force_unzip")
1032
+
1033
+ if "local_files_only" in dataset_kwargs:
1034
+ dataset_kwargs.pop("local_files_only")
1035
+
1036
+ if "create_link" in dataset_kwargs:
1037
+ dataset_kwargs.pop("create_link")
1038
+
1039
+ if dataset_kwargs is not None and "load_from_disk" in dataset_kwargs and dataset_kwargs["load_from_disk"]:
1040
+ dataset_kwargs.pop("load_from_disk")
1041
+ # using local task in offline environment, need to process the online dataset into local format via
1042
+ # `ds = load_datasets("lmms-lab/MMMU")`
1043
+ self.dataset = datasets.load_from_disk(path=self.DATASET_PATH, name=self.DATASET_NAME)
1044
+ else:
1045
+ self.dataset = datasets.load_dataset(
1046
+ path=self.DATASET_PATH,
1047
+ name=self.DATASET_NAME,
1048
+ download_mode=datasets.DownloadMode.REUSE_DATASET_IF_EXISTS,
1049
+ download_config=download_config,
1050
+ **dataset_kwargs if dataset_kwargs is not None else {},
1051
+ )
1052
+
1053
+ if self.config.process_docs is not None:
1054
+ for split in self.dataset:
1055
+ if split in [self.config.training_split, self.config.validation_split, self.config.test_split, self.config.fewshot_split]:
1056
+ self.dataset[split] = self.config.process_docs(self.dataset[split])
1057
+
1058
+ # copy dataset, remove image features
1059
+ self.dataset_no_image = self.dataset.copy()
1060
+ for doc_name in self.dataset_no_image:
1061
+ remove_cols = []
1062
+ features = self.dataset_no_image[doc_name].features
1063
+ # If it is an Image instance or a Sequence of Image instance. Remove it
1064
+ for feature in features:
1065
+ if isinstance(features[feature], Image):
1066
+ remove_cols.append(feature)
1067
+ elif isinstance(features[feature], Sequence) and isinstance(features[feature].feature, Image):
1068
+ remove_cols.append(feature)
1069
+ elif isinstance(features[feature], Audio):
1070
+ remove_cols.append(feature)
1071
+ for remove_col in remove_cols:
1072
+ self.dataset_no_image[doc_name] = self.dataset_no_image[doc_name].remove_columns(remove_col)
1073
+
1074
+ def has_training_docs(self) -> bool:
1075
+ if self.config.training_split is not None:
1076
+ return True
1077
+ else:
1078
+ return False
1079
+
1080
+ def has_validation_docs(self) -> bool:
1081
+ if self.config.validation_split is not None:
1082
+ return True
1083
+ else:
1084
+ return False
1085
+
1086
+ def has_test_docs(self) -> bool:
1087
+ if self.config.test_split is not None:
1088
+ return True
1089
+ else:
1090
+ return False
1091
+
1092
+ def training_docs(self) -> datasets.Dataset:
1093
+ if self.has_training_docs():
1094
+ return self.dataset[self.config.training_split]
1095
+
1096
+ def validation_docs(self) -> datasets.Dataset:
1097
+ if self.has_validation_docs():
1098
+ return self.dataset[self.config.validation_split]
1099
+
1100
+ def validation_docs_no_media(self) -> datasets.Dataset:
1101
+ if self.has_validation_docs():
1102
+ return self.dataset_no_image[self.config.validation_split]
1103
+
1104
+ def test_docs(self) -> datasets.Dataset:
1105
+ if self.has_test_docs():
1106
+ return self.dataset[self.config.test_split]
1107
+
1108
+ def test_docs_no_media(self) -> datasets.Dataset:
1109
+ if self.has_test_docs():
1110
+ return self.dataset_no_image[self.config.test_split]
1111
+
1112
+ @property
1113
+ def eval_docs_no_media(self) -> Union[datasets.Dataset, List[dict]]:
1114
+ if self.has_test_docs():
1115
+ return self.test_docs_no_media()
1116
+ elif self.has_validation_docs():
1117
+ return self.validation_docs_no_media()
1118
+ else:
1119
+ raise ValueError(f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!")
1120
+
1121
+ def fewshot_docs(self):
1122
+ if self.config.fewshot_split is not None:
1123
+ return self.dataset[self.config.fewshot_split]
1124
+ else:
1125
+ if (self.config.num_fewshot is not None) and (self.config.num_fewshot > 0):
1126
+ eval_logger.warning(f"Task '{self.config.task}': " "num_fewshot > 0 but fewshot_split is None. " "using preconfigured rule.")
1127
+ return super().fewshot_docs()
1128
+
1129
+ @utils.positional_deprecated
1130
+ def fewshot_context(
1131
+ self,
1132
+ doc: str,
1133
+ num_fewshot: int,
1134
+ system_instruction: Optional[str] = None,
1135
+ apply_chat_template: bool = False,
1136
+ fewshot_as_multiturn: bool = False,
1137
+ chat_template: Optional[Callable] = None,
1138
+ is_multimodal: bool = False,
1139
+ ) -> str:
1140
+ """Returns a fewshot context string that is made up of a prepended description
1141
+ (if provided), the `num_fewshot` number of examples, and an appended prompt example.
1142
+
1143
+ :param doc: str
1144
+ The document as returned from training_docs, validation_docs, or test_docs.
1145
+ :param num_fewshot: int
1146
+ The number of fewshot examples to provide in the returned context string.
1147
+ :param system_instruction: str
1148
+ System instruction to be applied to the prompt.
1149
+ :param apply_chat_template: bool
1150
+ Whether to apply the chat template to the fewshot context.
1151
+ :param fewshot_as_multiturn: bool
1152
+ Whether to provide the fewshot examples as a multiturn conversation or a single user turn.
1153
+ :param chat_template:
1154
+ callable (from lm.apply_chat_template) that takes in a list[Dict] chat transcript and renders it into a string.
1155
+ :returns: str
1156
+ The fewshot context.
1157
+ """
1158
+
1159
+ if apply_chat_template:
1160
+ labeled_examples = []
1161
+ else:
1162
+ labeled_examples = ""
1163
+
1164
+ # get task description
1165
+ if description := self.config.description:
1166
+ description = utils.apply_template(self.config.description, doc)
1167
+
1168
+ # create system prompt based on the provided system instruction and description
1169
+ if system_instruction is not None and description:
1170
+ system_prompt = f"{system_instruction}{self.sampler.fewshot_delimiter}{description}"
1171
+ elif system_instruction is not None:
1172
+ system_prompt = system_instruction
1173
+ elif description:
1174
+ system_prompt = description
1175
+ else:
1176
+ system_prompt = ""
1177
+
1178
+ # add system prompt if specified
1179
+ if system_prompt:
1180
+ if apply_chat_template:
1181
+ labeled_examples.append({"role": "system", "content": system_prompt})
1182
+ else:
1183
+ labeled_examples = system_prompt
1184
+
1185
+ # if few-shot - append examples after the system prompt
1186
+ if num_fewshot > 0:
1187
+ if is_multimodal is False:
1188
+ if apply_chat_template:
1189
+ labeled_examples.extend(self.sampler.get_chat_context(doc, num_fewshot, fewshot_as_multiturn))
1190
+ else:
1191
+ labeled_examples += self.sampler.get_context(doc, num_fewshot)
1192
+ else:
1193
+ if apply_chat_template:
1194
+ labeled_examples_text, labeled_examples_multimodal = self.sampler.get_multimodal_chat_context(doc, num_fewshot, fewshot_as_multiturn)
1195
+ labeled_examples.extend(labeled_examples_text)
1196
+ else:
1197
+ labeled_examples_text, labeled_examples_multimodal = self.sampler.get_multimodal_context(doc, num_fewshot)
1198
+ labeled_examples += labeled_examples_text
1199
+
1200
+ example = self.doc_to_text(doc)
1201
+ if is_multimodal is False:
1202
+ if apply_chat_template:
1203
+ if self.multiple_input:
1204
+ return chat_template(labeled_examples)
1205
+ if isinstance(example, str):
1206
+ self.append_target_question(labeled_examples, example, fewshot_as_multiturn)
1207
+ # for loglikelihood create a list of questions with appended choices
1208
+ elif isinstance(example, list):
1209
+ labeled_examples_list = []
1210
+ # copy chat history for each example and append the answer
1211
+ for ex in example:
1212
+ chat = copy.deepcopy(labeled_examples)
1213
+ self.append_target_question(chat, ex, fewshot_as_multiturn)
1214
+ labeled_examples_list.append(chat_template(chat))
1215
+ return labeled_examples_list
1216
+ # if example is an integer, append the choice or convert to string
1217
+ elif isinstance(example, int):
1218
+ if self.config.doc_to_choice is not None:
1219
+ choices = self.doc_to_choice(doc)
1220
+ self.append_target_question(labeled_examples, choices[example], fewshot_as_multiturn)
1221
+ else:
1222
+ self.append_target_question(labeled_examples, str(example), fewshot_as_multiturn)
1223
+ # return lm.apply_chat_template(labeled_examples)
1224
+ return chat_template(labeled_examples)
1225
+ else:
1226
+ if self.multiple_input:
1227
+ return labeled_examples
1228
+ if isinstance(example, str):
1229
+ return labeled_examples + example
1230
+ elif isinstance(example, list):
1231
+ return [labeled_examples + ex for ex in example]
1232
+ elif isinstance(example, int):
1233
+ if self.config.doc_to_choice is not None:
1234
+ choices = self.doc_to_choice(doc)
1235
+ return labeled_examples + choices[example]
1236
+ else:
1237
+ return labeled_examples + str(example)
1238
+ else:
1239
+ if apply_chat_template:
1240
+ raise NotImplementedError("Multimodal chat template not implemented yet")
1241
+ else:
1242
+ if self.multiple_input:
1243
+ return labeled_examples + "<image> " + example, labeled_examples_multimodal
1244
+ if isinstance(example, str):
1245
+ return labeled_examples + "<image> " + example, labeled_examples_multimodal
1246
+ else:
1247
+ raise NotImplementedError("Multimodal not implemented yet")
1248
+ # elif isinstance(example, list):
1249
+ # return [labeled_examples + ex for ex in example]
1250
+ # elif isinstance(example, int):
1251
+ # if self.config.doc_to_choice is not None:
1252
+ # choices = self.doc_to_choice(doc)
1253
+ # return labeled_examples + choices[example], labeled_examples_multimodal
1254
+ # else:
1255
+ # return labeled_examples + str(example), labeled_examples_multimodal
1256
+
1257
+ def apply_filters(self) -> Optional[List[Instance]]:
1258
+ """Iterates over FilterEnsembles and applies them to instances"""
1259
+ if hasattr(self, "_filters"):
1260
+ for f in self._filters:
1261
+ f.apply(self._instances, self.task_docs)
1262
+ else:
1263
+ eval_logger.warning("No filter defined, passing through instances")
1264
+ return self._instances
1265
+
1266
+ def should_decontaminate(self):
1267
+ return self.config.should_decontaminate
1268
+
1269
+ def doc_to_decontamination_query(self, doc):
1270
+ if self.config.should_decontaminate:
1271
+ if self.config.doc_to_decontamination_query is None:
1272
+ return self.doc_to_text(doc)
1273
+ else:
1274
+ doc_to_decontamination_query = self.config.doc_to_decontamination_query
1275
+ if doc_to_decontamination_query in self.features:
1276
+ return doc[doc_to_decontamination_query]
1277
+ elif callable(doc_to_decontamination_query):
1278
+ return doc_to_decontamination_query(doc)
1279
+ else:
1280
+ return ast.literal_eval(utils.apply_template(self.config.doc_to_decontamination_query, doc))
1281
+
1282
+ def _process_doc(self, doc):
1283
+ """
1284
+ Override this to process (detokenize, strip, replace, etc.) individual
1285
+ documents. This can be used in a map over documents of a data split.
1286
+ E.g. `map(self._process_doc, self.dataset["validation"])`
1287
+
1288
+ :return: dict
1289
+ The processed version of the specified `doc`.
1290
+ """
1291
+ return doc
1292
+
1293
+ def doc_to_text(self, doc):
1294
+ doc_to_text = self.config.doc_to_text
1295
+
1296
+ if type(doc_to_text) == int:
1297
+ return doc_to_text
1298
+ elif type(doc_to_text) == str:
1299
+ if doc_to_text in self.features:
1300
+ # if self.config.doc_to_choice is not None:
1301
+ # return self.doc_to_choice(doc)[doc[doc_to_text]]
1302
+ # else:
1303
+ return doc[doc_to_text]
1304
+ else:
1305
+ text_string = utils.apply_template(doc_to_text, doc)
1306
+ if text_string.isdigit() and self._config.doc_to_choice is not None:
1307
+ return ast.literal_eval(text_string)
1308
+ else:
1309
+ return text_string
1310
+ elif callable(doc_to_text):
1311
+ return (
1312
+ doc_to_text(doc, self.lmms_eval_specific_kwargs)
1313
+ if self.lmms_eval_specific_kwargs is not None
1314
+ else doc_to_text(
1315
+ doc,
1316
+ )
1317
+ )
1318
+ # Used when applying a Promptsource template
1319
+ elif hasattr(doc_to_text, "apply"):
1320
+ applied_prompt = doc_to_text.apply(doc)
1321
+ if len(applied_prompt) == 2:
1322
+ return applied_prompt[0]
1323
+ else:
1324
+ eval_logger.warning("Applied prompt returns empty string")
1325
+ return self.config.fewshot_delimiter
1326
+ else:
1327
+ print(type(doc_to_text))
1328
+ raise TypeError
1329
+
1330
+ def doc_to_target(self, doc: dict) -> Union[int, str, list]:
1331
+ doc_to_target = self.config.doc_to_target
1332
+
1333
+ if type(doc_to_target) == int:
1334
+ return doc_to_target
1335
+ elif type(doc_to_target) == str:
1336
+ if doc_to_target in self.features:
1337
+ # if self.config.doc_to_choice is not None:
1338
+ # return self.doc_to_choice(doc)[doc[doc_to_target]]
1339
+ # else:
1340
+ return doc[doc_to_target]
1341
+ else:
1342
+ target_string = utils.apply_template(doc_to_target, doc)
1343
+ if target_string.isdigit() and self._config.doc_to_choice is not None:
1344
+ return ast.literal_eval(target_string)
1345
+ elif len(target_string) >= 2 and (target_string[0] == "[") and (target_string[-1] == "]"):
1346
+ try:
1347
+ return ast.literal_eval(target_string)
1348
+ except (SyntaxError, ValueError):
1349
+ return target_string
1350
+ else:
1351
+ return target_string
1352
+ elif type(doc_to_target) == list:
1353
+ return doc_to_target
1354
+ elif callable(doc_to_target):
1355
+ return doc_to_target(doc, self.model_specific_target_kwargs) if self.model_specific_target_kwargs is not None else doc_to_target(doc)
1356
+ # Used when applying a Promptsource template
1357
+ elif hasattr(doc_to_target, "apply"):
1358
+ applied_prompt = doc_to_target.apply(doc)
1359
+ if len(applied_prompt) == 2:
1360
+ return applied_prompt[1]
1361
+ else:
1362
+ eval_logger.warning("Applied prompt returns empty string")
1363
+ return self.config.fewshot_delimiter
1364
+ else:
1365
+ raise TypeError
1366
+
1367
+ def doc_to_visual(self, doc: dict) -> Union[int, str, list]:
1368
+ self.config.doc_to_visual
1369
+ if type(self.config.doc_to_visual) == str:
1370
+ assert self.config.doc_to_visual in self.features
1371
+ # Single image. Still return a list for consistency.
1372
+ return [doc[self.config.doc_to_visual]]
1373
+ elif callable(self.config.doc_to_visual):
1374
+ return (
1375
+ self.config.doc_to_visual(doc, self.lmms_eval_specific_kwargs)
1376
+ if self.lmms_eval_specific_kwargs is not None and len(inspect.signature(self.config.doc_to_visual).parameters) == 2
1377
+ else self.config.doc_to_visual(
1378
+ doc,
1379
+ )
1380
+ )
1381
+ else:
1382
+ # eval_logger.warning("Note that doc_to_visual was called but not set in config. Please check if this is a text-only task.")
1383
+ return self.config.doc_to_visual
1384
+
1385
+ def doc_to_choice(self, doc: Any) -> List[str]:
1386
+ if self.config.doc_to_choice is None:
1387
+ eval_logger.error("Note that doc_to_choice was called but not set in config.")
1388
+ else:
1389
+ doc_to_choice = self.config.doc_to_choice
1390
+
1391
+ if type(doc_to_choice) == str:
1392
+ if doc_to_choice in self.features:
1393
+ return doc[doc_to_choice]
1394
+ else:
1395
+ return ast.literal_eval(utils.apply_template(doc_to_choice, doc))
1396
+ elif type(doc_to_choice) == list:
1397
+ return doc_to_choice
1398
+ elif type(doc_to_choice) == dict:
1399
+ return list(doc_to_choice.values())
1400
+ elif callable(doc_to_choice):
1401
+ return doc_to_choice(doc)
1402
+ elif hasattr(doc_to_choice, "get_answer_choices_list"):
1403
+ return doc_to_choice.get_answer_choices_list(doc)
1404
+ else:
1405
+ raise TypeError
1406
+
1407
+ def construct_requests(self, doc_id: int, ctx: str, **kwargs) -> Union[List[Instance], Instance]:
1408
+ split = kwargs.get("metadata").get("split")
1409
+ # kwargs.pop("split")
1410
+ if self.OUTPUT_TYPE == "loglikelihood":
1411
+ arguments = (ctx, self.doc_to_target, self.doc_to_visual, doc_id, self.config.task, split)
1412
+ elif self.OUTPUT_TYPE == "multiple_choice":
1413
+ doc = self.dataset[split][doc_id]
1414
+ choices = self.doc_to_choice(doc)
1415
+ target_delimiter = self.config.target_delimiter
1416
+ if self.multiple_input:
1417
+ # If there are multiple inputs, choices are placed in the ctx
1418
+ cont = self.doc_to_target(doc)
1419
+ arguments = [(ctx, f"{target_delimiter}{cont}", self.doc_to_visual, doc_id, self.config.task, split) for ctx in choices]
1420
+ else:
1421
+ # Otherwise they are placed in the continuation
1422
+ arguments = [(ctx, f"{target_delimiter}{cont}", self.doc_to_visual, doc_id, self.config.task, split) for cont in choices]
1423
+ request_list = [
1424
+ Instance(
1425
+ request_type="loglikelihood",
1426
+ # doc=doc,
1427
+ arguments=arg,
1428
+ idx=i,
1429
+ **kwargs,
1430
+ )
1431
+ for i, arg in enumerate(arguments)
1432
+ ]
1433
+ # TODO: we should raise a warning telling users this will at most ~2x runtime.
1434
+ if "acc_mutual_info" in self._metric_fn_list.keys():
1435
+ # if we are calculating multiple choice accuracy
1436
+ # using mutual information instead of raw loglikelihood as metric, need unconditional lls.
1437
+
1438
+ # here mutual info refers to calculating
1439
+ # log(P(choice|ctx) / P(choice)) = log(P(choice|ctx)) - log(P(choice))
1440
+ # in other words normalizing by subtracting the unconditional logprob of each choice.
1441
+ request_list.extend(
1442
+ [
1443
+ Instance(
1444
+ request_type="loglikelihood",
1445
+ # doc=doc,
1446
+ arguments=("", "{}".format(choice)),
1447
+ idx=i,
1448
+ **kwargs,
1449
+ )
1450
+ for i, choice in enumerate(choices)
1451
+ ]
1452
+ )
1453
+ return request_list
1454
+
1455
+ elif self.OUTPUT_TYPE == "generate_until":
1456
+ arguments = (ctx, copy.deepcopy(self.config.generation_kwargs), self.doc_to_visual, doc_id, self.config.task, split)
1457
+ elif self.OUTPUT_TYPE == "generate_until_multi_round":
1458
+ arguments = (ctx, copy.deepcopy(self.config.generation_kwargs), self.doc_to_visual, partial(self.config.doc_to_text, lmms_eval_specific_kwargs=self.lmms_eval_specific_kwargs), doc_id, self.config.task, split)
1459
+ return Instance(request_type=self.OUTPUT_TYPE, arguments=arguments, idx=0, **kwargs)
1460
+
1461
+ # TODO: we add a full_docs interface here for some evaluations that needs to access the full datasets during process_results function. we may have better ways to handle this.
1462
+ @retry(stop=(stop_after_attempt(5) | stop_after_delay(1200)), wait=wait_fixed(2))
1463
+ def process_results(self, doc, results, full_docs=None):
1464
+ if self.OUTPUT_TYPE == "generate_until":
1465
+ if isinstance(results, list) and isinstance(results[0], list):
1466
+ results = [res.strip() for res in results[0]]
1467
+ else:
1468
+ results = [res.strip() for res in results]
1469
+
1470
+ kwargs = {}
1471
+ if full_docs is not None:
1472
+ kwargs["full_docs"] = full_docs
1473
+ if callable(self.config.process_results):
1474
+ return self.config.process_results(doc, results, **kwargs)
1475
+
1476
+ result_dict = {}
1477
+ use_metric = list(self._metric_fn_list.keys())
1478
+ if self.OUTPUT_TYPE == "loglikelihood":
1479
+ ll, is_greedy = results
1480
+ return {
1481
+ **({"perplexity": ll} if "perplexity" in use_metric else {}),
1482
+ **({"acc": int(is_greedy)} if "acc" in use_metric else {}),
1483
+ }
1484
+ elif self.OUTPUT_TYPE == "multiple_choice":
1485
+ lls, is_greedy = zip(*results)
1486
+
1487
+ # retrieve choices in List[str] form, to compute choice lengths, etc.
1488
+ choices = self.doc_to_choice(doc)
1489
+ completion_len = np.array([float(len(i)) for i in choices])
1490
+
1491
+ if 2 * len(choices) == len(lls) and "acc_mutual_info" in self._metric_fn_list.keys():
1492
+ # then we are doing mutual info.
1493
+ # this stores the "dryrun" / unconditional answer loglikelihoods
1494
+ lls_unconditional = lls[1::2]
1495
+ assert len(lls_unconditional) == len(choices)
1496
+ # and this stores our "regular" conditional loglikelihoods
1497
+ lls = lls[::2]
1498
+
1499
+ # Warning :
1500
+ # Here may be different from original lm-eval
1501
+ # since we return the actual loss in many model loglikelihood
1502
+ # we just use the argmin here
1503
+ pred = np.argmin(lls)
1504
+ pred_norm = np.argmin(lls / completion_len)
1505
+
1506
+ if self.multiple_input:
1507
+ gold = self.doc_to_text(doc)
1508
+ else:
1509
+ gold = self.doc_to_target(doc)
1510
+
1511
+ gold_index_error = False
1512
+ if type(gold) is list:
1513
+ gold = [i if i < len(choices) else -100 for i in gold]
1514
+ if -100 in gold:
1515
+ gold_index_error = True
1516
+ else:
1517
+ if type(gold) is int:
1518
+ gold = gold if gold < len(choices) else -100
1519
+ elif type(gold) is str:
1520
+ gold = choices.index(gold) if gold in choices else -100
1521
+
1522
+ if gold == -100:
1523
+ gold_index_error = True
1524
+
1525
+ if gold_index_error:
1526
+ eval_logger.warning(f"Label index was not in within range of available choices," f"Sample:\n\n{doc}\n\n")
1527
+
1528
+ if self.multiple_target:
1529
+ acc = 1.0 if pred in gold else 0.0
1530
+ acc_norm = 1.0 if pred_norm in gold else 0.0
1531
+ exact_match = int(any([is_greedy[i] if i != -100 else 0 for i in gold]))
1532
+ else:
1533
+ acc = 1.0 if pred == gold else 0.0
1534
+ acc_norm = 1.0 if pred_norm == gold else 0.0
1535
+ # TODO: this gets score of 0 on arc_challenge for pythia-70m. need to test that this works properly
1536
+ exact_match = int(is_greedy[gold]) if gold != -100 else 0
1537
+
1538
+ result_dict = {
1539
+ **({"acc": acc} if "acc" in use_metric else {}),
1540
+ **({"f1": (gold, pred)} if "f1" in use_metric else {}),
1541
+ **({"mcc": (gold, pred)} if "mcc" in use_metric else {}),
1542
+ **({"acc_norm": acc_norm} if "acc_norm" in use_metric else {}),
1543
+ **({"exact_match": exact_match} if "exact_match" in use_metric else {}),
1544
+ }
1545
+
1546
+ if "acc_mutual_info" in use_metric:
1547
+ lls_mutual_info = [ll_c - ll_u for ll_c, ll_u in zip(lls, lls_unconditional)]
1548
+ acc_mutual_info = 1.0 if np.argmax(lls_mutual_info) == gold else 0.0
1549
+ result_dict["acc_mutual_info"] = acc_mutual_info
1550
+
1551
+ elif "generate_until" in self.OUTPUT_TYPE:
1552
+ gold = self.doc_to_target(doc)
1553
+ result = [res.strip() for res in results]
1554
+ if self.config.doc_to_choice is not None:
1555
+ # If you set doc_to_choice,
1556
+ # it assumes that doc_to_target returns a number.
1557
+ choices = self.doc_to_choice(doc)
1558
+ gold = choices[gold]
1559
+ # we expect multiple_targets to be a list.
1560
+ elif self.multiple_target:
1561
+ gold = list(gold)
1562
+ # elif type(gold) != type(result):
1563
+ # # cast gold to the same type as result
1564
+ # gold = type(result)(gold)
1565
+
1566
+ for metric in self._metric_fn_list.keys():
1567
+ if self.multiple_target and metric != "anls":
1568
+ # in the case where we have multiple targets,
1569
+ # return true if any are true
1570
+ # TODO: this may break for multipLe_target, non zero-or-1 metrics
1571
+ scores = []
1572
+ if not isinstance(gold, list):
1573
+ # sometimes, a multiple_target dataset has exceptions where one doc has only one string answer
1574
+ # print(gold)
1575
+ gold = [gold]
1576
+ for gold_option in gold:
1577
+ try:
1578
+ result_score = self._metric_fn_list[metric](
1579
+ references=[gold_option],
1580
+ predictions=result,
1581
+ **self._metric_fn_kwargs[metric],
1582
+ )
1583
+ except TypeError: # TODO: this is hacky and I don't want to do it
1584
+ result_score = self._metric_fn_list[metric]([gold_option, result])
1585
+ if isinstance(result_score, dict):
1586
+ # TODO: this handles the case where HF evaluate returns a dict.
1587
+ result_score = result_score[metric]
1588
+ scores.append(result_score)
1589
+ if any(scores):
1590
+ result_score = 1.0
1591
+ else:
1592
+ result_score = 0.0
1593
+ else:
1594
+ if not isinstance(gold, list):
1595
+ gold = [gold]
1596
+ try:
1597
+ result_score = self._metric_fn_list[metric](
1598
+ references=gold,
1599
+ predictions=result,
1600
+ **self._metric_fn_kwargs[metric],
1601
+ )
1602
+ except TypeError: # needed for now in order to use a different interface between our own metrics and HF Evaluate metrics
1603
+ result_score = self._metric_fn_list[metric]([gold, result])
1604
+ if isinstance(result_score, dict):
1605
+ # TODO: this handles the case where HF evaluate returns a dict.
1606
+ result_score = result_score[metric]
1607
+ result_dict[metric] = result_score
1608
+ else:
1609
+ raise ValueError(
1610
+ f"Passed invalid output_type '{self.OUTPUT_TYPE}' ! Please use one of ",
1611
+ "'loglikelihood','generate_until', 'generate_until_multi_round', or 'multiple_choice'",
1612
+ )
1613
+
1614
+ return result_dict
1615
+
1616
+ def aggregation(self):
1617
+ return self._aggregation_list
1618
+
1619
+ def higher_is_better(self):
1620
+ return self._higher_is_better
1621
+
1622
+ def get_config(self, key: str) -> Any:
1623
+ return getattr(self._config, key, None)
1624
+
1625
+ @property
1626
+ def task_name(self) -> Any:
1627
+ return getattr(self.config, "task", None)
1628
+
1629
+ def __repr__(self):
1630
+ return f"ConfigurableTask(task_name={getattr(self.config, 'task', None)}," f"output_type={self.OUTPUT_TYPE}," f"num_fewshot={getattr(self.config, 'num_fewshot', None)}," f"num_samples={len(self.eval_docs)})"
LaViDa/eval/lmms_eval/caching/__init__.py ADDED
File without changes
LaViDa/eval/lmms_eval/caching/cache.py ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import hashlib
2
+ import os
3
+
4
+ import dill
5
+
6
+ from lmms_eval.loggers.utils import _handle_non_serializable
7
+ from lmms_eval.utils import eval_logger
8
+
9
+ MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
10
+
11
+ OVERRIDE_PATH = os.getenv("LM_HARNESS_CACHE_PATH")
12
+
13
+
14
+ PATH = OVERRIDE_PATH if OVERRIDE_PATH else f"{MODULE_DIR}/.cache"
15
+
16
+ # This should be sufficient for uniqueness
17
+ HASH_INPUT = "EleutherAI-lm-evaluation-harness"
18
+
19
+ HASH_PREFIX = hashlib.sha256(HASH_INPUT.encode("utf-8")).hexdigest()
20
+
21
+ FILE_SUFFIX = f".{HASH_PREFIX}.pickle"
22
+
23
+
24
+ def load_from_cache(file_name):
25
+ try:
26
+ path = f"{PATH}/{file_name}{FILE_SUFFIX}"
27
+
28
+ with open(path, "rb") as file:
29
+ cached_task_dict = dill.loads(file.read())
30
+ return cached_task_dict
31
+
32
+ except Exception:
33
+ eval_logger.debug(f"{file_name} is not cached, generating...")
34
+ pass
35
+
36
+
37
+ def save_to_cache(file_name, obj):
38
+ if not os.path.exists(PATH):
39
+ os.mkdir(PATH)
40
+
41
+ file_path = f"{PATH}/{file_name}{FILE_SUFFIX}"
42
+
43
+ serializable_obj = []
44
+
45
+ for item in obj:
46
+ sub_serializable_obj = []
47
+ for subitem in item:
48
+ if hasattr(subitem, "arguments"): # we need to handle the arguments specially since doc_to_visual is callable method and not serializable
49
+ serializable_arguments = tuple(arg if not callable(arg) else None for arg in subitem.arguments)
50
+ subitem.arguments = serializable_arguments
51
+ sub_serializable_obj.append(_handle_non_serializable(subitem))
52
+ serializable_obj.append(sub_serializable_obj)
53
+
54
+ eval_logger.debug(f"Saving {file_path} to cache...")
55
+ with open(file_path, "wb") as file:
56
+ file.write(dill.dumps(serializable_obj))
57
+
58
+
59
+ # NOTE the "key" param is to allow for flexibility
60
+ def delete_cache(key: str = ""):
61
+ files = os.listdir(PATH)
62
+
63
+ for file in files:
64
+ if file.startswith(key) and file.endswith(FILE_SUFFIX):
65
+ file_path = f"{PATH}/{file}"
66
+ os.unlink(file_path)
LaViDa/eval/lmms_eval/eval_server.py ADDED
File without changes
LaViDa/eval/lmms_eval/evaluator.py ADDED
@@ -0,0 +1,654 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import collections
2
+ import inspect
3
+ import itertools
4
+ import json
5
+ import os
6
+ import random
7
+ import sys
8
+ import time
9
+ from collections import defaultdict
10
+ from dataclasses import dataclass
11
+ from typing import List, Optional, Union
12
+
13
+ import numpy as np
14
+ import torch
15
+ import torch.distributed as dist
16
+ from datasets import Image, Sequence
17
+ from loguru import logger as eval_logger
18
+ from tqdm import tqdm
19
+
20
+ import lmms_eval.api
21
+ import lmms_eval.api.metrics
22
+ import lmms_eval.api.registry
23
+ from lmms_eval.evaluator_utils import (
24
+ consolidate_group_results,
25
+ consolidate_results,
26
+ get_sample_size,
27
+ get_subtask_list,
28
+ get_task_list,
29
+ prepare_print_tasks,
30
+ print_writeout,
31
+ run_task_tests,
32
+ )
33
+ from lmms_eval.loggers.evaluation_tracker import EvaluationTracker
34
+ from lmms_eval.models import get_model
35
+ from lmms_eval.tasks import TaskManager, get_task_dict
36
+ from lmms_eval.utils import (
37
+ create_iterator,
38
+ get_datetime_str,
39
+ get_git_commit_hash,
40
+ handle_non_serializable,
41
+ hash_string,
42
+ make_table,
43
+ positional_deprecated,
44
+ run_task_tests,
45
+ simple_parse_args_string,
46
+ )
47
+
48
+
49
+ @positional_deprecated
50
+ def simple_evaluate(
51
+ model,
52
+ model_args: Optional[Union[str, dict]] = None,
53
+ tasks: Optional[List[Union[str, dict, object]]] = None,
54
+ num_fewshot: Optional[int] = None,
55
+ batch_size: Optional[Union[int, str]] = None,
56
+ max_batch_size: Optional[int] = None,
57
+ device: Optional[str] = None,
58
+ use_cache: Optional[str] = None,
59
+ cache_requests: bool = False,
60
+ rewrite_requests_cache: bool = False,
61
+ delete_requests_cache: bool = False,
62
+ limit: Optional[Union[int, float]] = None,
63
+ bootstrap_iters: int = 100000,
64
+ check_integrity: bool = False,
65
+ write_out: bool = False,
66
+ log_samples: bool = True,
67
+ evaluation_tracker: Optional[EvaluationTracker] = None,
68
+ system_instruction: Optional[str] = None,
69
+ apply_chat_template: bool = False,
70
+ fewshot_as_multiturn: bool = False,
71
+ gen_kwargs: Optional[str] = None,
72
+ task_manager: Optional[TaskManager] = None,
73
+ verbosity: str = "INFO",
74
+ predict_only: bool = False,
75
+ random_seed: int = 0,
76
+ numpy_random_seed: int = 1234,
77
+ torch_random_seed: int = 1234,
78
+ fewshot_random_seed: int = 1234,
79
+ datetime_str: str = get_datetime_str(),
80
+ cli_args=None,
81
+ ):
82
+ """Instantiate and evaluate a model on a list of tasks.
83
+
84
+ :param model: Union[str, LM]
85
+ Name of model or LM object, see lm_eval.models.get_model
86
+ :param model_args: Optional[str, dict]
87
+ String or dict arguments for each model class, see LM.create_from_arg_string and LM.create_from_arg_object.
88
+ Ignored if `model` argument is a LM object.
89
+ :param tasks: list[Union[str, dict, Task]]
90
+ List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.
91
+ :param num_fewshot: int
92
+ Number of examples in few-shot context
93
+ :param batch_size: int or str, optional
94
+ Batch size for model
95
+ :param max_batch_size: int, optional
96
+ Maximal batch size to try with automatic batch size detection
97
+ :param device: str, optional
98
+ PyTorch device (e.g. "cpu" or "cuda:0") for running models
99
+ :param use_cache: str, optional
100
+ A path to a sqlite db file for caching model responses. `None` if not caching.
101
+ :param cache_requests: bool, optional
102
+ Speed up evaluation by caching the building of dataset requests. `None` if not caching.
103
+ :param rewrite_requests_cache: bool, optional
104
+ Rewrites all of the request cache if set to `True`. `None` if not desired.
105
+ :param delete_requests_cache: bool, optional
106
+ Deletes all of the request cache if set to `True`. `None` if not desired.
107
+ :param limit: int or float, optional
108
+ Limit the number of examples per task (only use this for testing), If <1, limit is a percentage of the total number of examples.
109
+ :param bootstrap_iters:
110
+ Number of iterations for bootstrap statistics, used when calculating stderrs. set to 0 for no stderr calculations to be performed.
111
+ :param check_integrity: bool
112
+ Whether to run the relevant part of the test suite for the tasks
113
+ :param write_out: bool
114
+ If True, write out an example document and model input for checking task integrity
115
+ :param log_samples: bool
116
+ If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis
117
+ :param system_instruction: str
118
+ System instruction to be applied to the prompt
119
+ :param apply_chat_template: bool
120
+ If True, apply chat template to the prompt
121
+ :param fewshot_as_multiturn: bool
122
+ Whether to provide the fewshot examples as a multiturn conversation or a single user turn.
123
+ :param gen_kwargs: str
124
+ String arguments for model generation
125
+ Ignored for all tasks with loglikelihood output_type
126
+ :param predict_only: bool
127
+ If true only model outputs will be generated and returned. Metrics will not be evaluated
128
+ :param random_seed: int
129
+ Random seed for python's random module. If set to None, the seed will not be set.
130
+ :param numpy_random_seed: int
131
+ Random seed for numpy. If set to None, the seed will not be set.
132
+ :param torch_random_seed: int
133
+ Random seed for torch. If set to None, the seed will not be set.
134
+ :param fewshot_random_seed: int
135
+ Random seed for fewshot sampler random generator. If set to None, the seed of generator will be set to None.
136
+
137
+ :return
138
+ Dictionary of results
139
+ """
140
+ seed_message = []
141
+ if random_seed is not None:
142
+ # See https://github.com/EleutherAI/lm-evaluation-harness/pull/1412
143
+ seed_message.append(f"Setting random seed to {random_seed}")
144
+ random.seed(random_seed)
145
+
146
+ if numpy_random_seed is not None:
147
+ seed_message.append(f"Setting numpy seed to {numpy_random_seed}")
148
+ np.random.seed(numpy_random_seed)
149
+
150
+ if torch_random_seed is not None:
151
+ seed_message.append(f"Setting torch manual seed to {torch_random_seed}")
152
+ torch.manual_seed(torch_random_seed)
153
+
154
+ if seed_message:
155
+ eval_logger.info(" | ".join(seed_message))
156
+
157
+ assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names."
158
+
159
+ if gen_kwargs:
160
+ gen_kwargs = simple_parse_args_string(gen_kwargs)
161
+ eval_logger.warning(f"generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.")
162
+ if gen_kwargs == "":
163
+ gen_kwargs = None
164
+
165
+ if model_args is None:
166
+ model_args = ""
167
+
168
+ if task_manager is None:
169
+ task_manager = TaskManager(verbosity, model_name=model)
170
+
171
+ task_dict = get_task_dict(tasks, task_manager)
172
+
173
+ if isinstance(model, str):
174
+ if model_args is None:
175
+ model_args = ""
176
+ lm = lmms_eval.models.get_model(model).create_from_arg_string(
177
+ model_args,
178
+ {
179
+ "batch_size": batch_size,
180
+ "max_batch_size": max_batch_size,
181
+ "device": device,
182
+ },
183
+ )
184
+ elif isinstance(model, lmms_eval.api.model.lmms):
185
+ lm = model
186
+
187
+ # helper function to recursively apply config overrides to leaf subtasks, skipping their constituent groups.
188
+ # (setting of num_fewshot ; bypassing metric calculation ; setting fewshot seed)
189
+ def _adjust_config(task_dict):
190
+ adjusted_task_dict = {}
191
+ for task_name, task_obj in task_dict.items():
192
+ if isinstance(task_obj, dict):
193
+ adjusted_task_dict = {
194
+ **adjusted_task_dict,
195
+ **{task_name: _adjust_config(task_obj)},
196
+ }
197
+
198
+ else:
199
+ task_obj = task_dict[task_name]
200
+ if type(task_obj) == tuple:
201
+ group, task_obj = task_obj
202
+ if task_obj is None:
203
+ continue
204
+ lm.task_dict[task_name] = task_obj.dataset
205
+ if "generate_until" in task_obj.get_config("output_type"):
206
+ if gen_kwargs is not None:
207
+ task_obj.set_config(key="generation_kwargs", value=gen_kwargs, update=True)
208
+
209
+ if predict_only:
210
+ eval_logger.info(f"Processing {task_name} in output-only mode. Metrics will not be calculated!")
211
+ # we have to change the class properties post-hoc. This is pretty hacky.
212
+ task_obj.override_metric(metric_name="bypass")
213
+
214
+ # override tasks' fewshot values to the provided num_fewshot arg value
215
+ # except if tasks have it set to 0 manually in their configs--then we should never overwrite that
216
+ if num_fewshot is not None:
217
+ if (default_num_fewshot := task_obj.get_config("num_fewshot")) == 0:
218
+ eval_logger.info(f"num_fewshot has been set to 0 for {task_name} in its config. Manual configuration will be ignored.")
219
+ else:
220
+ eval_logger.warning(f"Overwriting default num_fewshot of {task_name} from {default_num_fewshot} to {num_fewshot}")
221
+ task_obj.set_config(key="num_fewshot", value=num_fewshot)
222
+ else:
223
+ # if num_fewshot not provided, and the task does not define a default one, default to 0
224
+ if (default_num_fewshot := task_obj.get_config("num_fewshot")) is None:
225
+ task_obj.set_config(key="num_fewshot", value=0)
226
+ # fewshot_random_seed set for tasks, even with a default num_fewshot (e.g. in the YAML file)
227
+ task_obj.set_fewshot_seed(seed=fewshot_random_seed)
228
+ # eval_logger.info(f"Setting fewshot random generator seed to {fewshot_random_seed}")
229
+
230
+ adjusted_task_dict[task_name] = task_obj
231
+
232
+ return adjusted_task_dict
233
+
234
+ task_dict = _adjust_config(task_dict)
235
+
236
+ if check_integrity:
237
+ run_task_tests(task_list=tasks)
238
+
239
+ if evaluation_tracker is not None:
240
+ evaluation_tracker.general_config_tracker.log_experiment_args(
241
+ model_source=model,
242
+ model_args=model_args,
243
+ system_instruction=system_instruction,
244
+ chat_template=lm.chat_template if apply_chat_template else None,
245
+ fewshot_as_multiturn=fewshot_as_multiturn,
246
+ )
247
+
248
+ results = evaluate(
249
+ lm=lm,
250
+ task_dict=task_dict,
251
+ limit=limit,
252
+ cache_requests=cache_requests,
253
+ rewrite_requests_cache=rewrite_requests_cache,
254
+ bootstrap_iters=bootstrap_iters,
255
+ write_out=write_out,
256
+ log_samples=True if predict_only else log_samples,
257
+ system_instruction=system_instruction,
258
+ apply_chat_template=apply_chat_template,
259
+ fewshot_as_multiturn=fewshot_as_multiturn,
260
+ verbosity=verbosity,
261
+ cli_args=cli_args,
262
+ )
263
+
264
+ if lm.rank == 0:
265
+ if isinstance(model, str):
266
+ model_name = model
267
+ elif hasattr(model, "config") and hasattr(model.config, "_name_or_path"):
268
+ model_name = model.config._name_or_path
269
+ else:
270
+ model_name = type(model).__name__
271
+
272
+ # add info about the model and few shot config
273
+ results["config"] = {
274
+ "model": model_name,
275
+ "model_args": model_args,
276
+ }
277
+ # add more detailed model info if available TODO: add model info
278
+ # if isinstance(lm, lm_eval.models.huggingface.HFLM):
279
+ # results["config"].update(lm.get_model_info())
280
+ # add info about execution
281
+ results["config"].update(
282
+ {
283
+ "batch_size": batch_size,
284
+ "batch_sizes": (list(lm.batch_sizes.values()) if hasattr(lm, "batch_sizes") else []),
285
+ "device": device,
286
+ "use_cache": use_cache,
287
+ "limit": limit,
288
+ "bootstrap_iters": bootstrap_iters,
289
+ "gen_kwargs": gen_kwargs,
290
+ "random_seed": random_seed,
291
+ "numpy_seed": numpy_random_seed,
292
+ "torch_seed": torch_random_seed,
293
+ "fewshot_seed": fewshot_random_seed,
294
+ }
295
+ )
296
+ results["git_hash"] = get_git_commit_hash()
297
+ results["date"] = datetime_str
298
+ # add_env_info(results) # additional environment info to results
299
+ # add_tokenizer_info(results, lm) # additional info about tokenizer
300
+ return results
301
+ else:
302
+ return None
303
+
304
+
305
+ decontaminate_suffix = "_decontaminate"
306
+
307
+
308
+ @positional_deprecated
309
+ def evaluate(
310
+ lm: "LM",
311
+ task_dict,
312
+ limit: Optional[int] = None,
313
+ cache_requests: bool = False,
314
+ rewrite_requests_cache: bool = False,
315
+ bootstrap_iters: Optional[int] = 100000,
316
+ write_out: bool = False,
317
+ log_samples: bool = True,
318
+ system_instruction: Optional[str] = None,
319
+ apply_chat_template: bool = False,
320
+ fewshot_as_multiturn: bool = False,
321
+ verbosity: str = "INFO",
322
+ cli_args=None,
323
+ ):
324
+ """Instantiate and evaluate a model on a list of tasks.
325
+
326
+ :param lm: obj
327
+ Language Model
328
+ :param task_dict: dict[str, Task]
329
+ Dictionary of tasks. Tasks will be taken to have name type(task).config.task .
330
+ :param limit: int, optional
331
+ Limit the number of examples per task (only use this for testing)
332
+ :param bootstrap_iters:
333
+ Number of iterations for bootstrap statistics, used when calculating stderr. Set to 0 for skipping all stderr calculations.
334
+ :param write_out: bool
335
+ If True, write out an example document and model input for checking task integrity
336
+ :param log_samples: bool
337
+ If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis
338
+ :param system_instruction: str
339
+ System instruction to be applied to the prompt
340
+ :param apply_chat_template: bool
341
+ If True, apply chat template to the prompt
342
+ :param fewshot_as_multiturn: bool
343
+ Whether to provide the fewshot examples as a multiturn conversation or a single user turn.
344
+ :return
345
+ Dictionary of results
346
+ """
347
+
348
+ # stores the final result for each task, for each metric/filter pair.
349
+ results = collections.defaultdict(dict)
350
+ # Tracks each task's version.
351
+ versions = collections.defaultdict(dict)
352
+ # Tracks the YAML configs of all chosen tasks.
353
+ configs = collections.defaultdict(dict)
354
+ # logs info about each document evaluated.
355
+ samples = collections.defaultdict(list)
356
+ # tracks all Instances/requests a model must generate output on.
357
+ requests = collections.defaultdict(list)
358
+ # Aggregated task scores presented with groups
359
+ results_agg = collections.defaultdict(dict)
360
+ # Aggregated groups scores only
361
+ groups_agg = collections.defaultdict(dict)
362
+ # stores the amount to pad out reqs per req. type so that
363
+ # number of fwd passes per distributed rank is equal
364
+ padding_requests = collections.defaultdict(int)
365
+ # store the hierarchy to do proper ordering
366
+ task_hierarchy = collections.defaultdict(list)
367
+ # store the ordering of tasks and groups
368
+ task_order = collections.defaultdict(int)
369
+ task_group_alias = collections.defaultdict(dict)
370
+ # store num-fewshot value per task
371
+ num_fewshot = collections.defaultdict(int)
372
+
373
+ # get lists of group hierarchy and each type of request
374
+ eval_tasks = get_task_list(task_dict)
375
+ name_to_task = {}
376
+ if not log_samples:
377
+ if not all("bypass" not in getattr(task_output.task, "_metric_fn_list", {}).keys() for task_output in eval_tasks):
378
+ raise ValueError("log_samples must be True for 'bypass' metric-only tasks")
379
+
380
+ for task_output in eval_tasks:
381
+ task: Task = task_output.task
382
+ task_name = task_output.task_name
383
+ task.args = cli_args
384
+
385
+ name_to_task[task_name] = task
386
+
387
+ if type(task) == tuple:
388
+ group_name, task = task
389
+ task_hierarchy[group_name].append(task_name)
390
+ versions[group_name] = "N/A"
391
+ else:
392
+ group_name = None
393
+ task_hierarchy[task_name] = []
394
+
395
+ if task is None:
396
+ continue
397
+
398
+ versions[task_name] = task.VERSION
399
+ configs[task_name] = dict(task.dump_config())
400
+
401
+ if "num_fewshot" in configs[task_name]:
402
+ n_shot = configs[task_name]["num_fewshot"]
403
+ else:
404
+ n_shot = 0
405
+ num_fewshot[task_name] = n_shot
406
+
407
+ if "task_alias" in configs[task_name]:
408
+ task_group_alias[task_name] = configs[task_name]["task_alias"]
409
+
410
+ if ("group_alias" in configs[task_name]) and (group_name not in task_group_alias) and (group_name is not None):
411
+ task_group_alias[group_name] = configs[task_name]["group_alias"]
412
+
413
+ limit = get_sample_size(task, limit)
414
+ task.build_all_requests(
415
+ limit=limit,
416
+ rank=lm.rank,
417
+ world_size=lm.world_size,
418
+ cache_requests=cache_requests, # later we will add them
419
+ rewrite_requests_cache=rewrite_requests_cache,
420
+ system_instruction=system_instruction,
421
+ apply_chat_template=apply_chat_template,
422
+ fewshot_as_multiturn=fewshot_as_multiturn,
423
+ chat_template=getattr(lm, "apply_chat_template") if apply_chat_template else None,
424
+ tokenizer_name=getattr(lm, "tokenizer_name", "") if apply_chat_template else "",
425
+ )
426
+ eval_logger.debug(f"Task: {task_output.task_name}; number of requests on this rank: {len(task._instances)}")
427
+ if write_out:
428
+ print_writeout(task)
429
+ # aggregate Instances by LM method requested to get output.
430
+ for instance in task.instances:
431
+ reqtype = instance.request_type
432
+ requests[reqtype].append(instance)
433
+
434
+ if lm.world_size > 1:
435
+ instances_rnk = torch.tensor(len(task._instances), device=lm.device)
436
+ gathered_item = lm.accelerator.gather(instances_rnk).cpu().detach().numpy().tolist()
437
+ # "multiple_choice" task types dispatch (several) "loglikelihood" request types
438
+ reqtype = "loglikelihood" if task.OUTPUT_TYPE == "multiple_choice" else task.OUTPUT_TYPE
439
+ # compute number of pseudo-batches to pad with (FSDP/DDP require even batches among ranks)
440
+ numpad = max(gathered_item) - gathered_item[lm.rank]
441
+ # todo: may not account for padding in cases like SquadV2 which has multiple req types
442
+ padding_requests[reqtype] += numpad
443
+
444
+ ### Run LMM on inputs, get all outputs ###
445
+ # execute each type of request
446
+ for reqtype, reqs in requests.items():
447
+ eval_logger.info("Running {} requests".format(reqtype))
448
+ # create `K` copies of each request `req` based off `K = req.repeats`
449
+ cloned_reqs = []
450
+ for req in reqs:
451
+ cloned_reqs.extend([req] * req.repeats)
452
+
453
+ if (lm.world_size > 1) and (padding_requests[reqtype] > 0):
454
+ for _ in range(padding_requests[reqtype]):
455
+ cloned_reqs.extend([req] * req.repeats)
456
+
457
+ # run requests through model
458
+ resps = getattr(lm, reqtype)(cloned_reqs) # Choiszt run generate until
459
+
460
+ # put responses from model into a list of length K for each request.
461
+ for x, req in zip(resps, cloned_reqs):
462
+ req.resps.append(x)
463
+
464
+ if lm.world_size > 1:
465
+ lm.accelerator.wait_for_everyone()
466
+
467
+ RANK = lm.rank
468
+ WORLD_SIZE = lm.world_size
469
+ ### Postprocess outputs ###
470
+ # TODO: del model here, maybe (idea: allow user to specify device of e.g. reward model separately)
471
+ for task_output in eval_tasks:
472
+ task = task_output.task
473
+ task.apply_filters()
474
+
475
+ ### Collect values of metrics on all datapoints ###
476
+ # # unpack results and sort back in order and return control to Task
477
+ # TODO: make it possible to use a different metric per filter
478
+ # Pre-process task.instances to group by doc_id
479
+ instances_by_doc_id = collections.defaultdict(list)
480
+ for instance in task.instances:
481
+ instances_by_doc_id[instance.doc_id].append(instance)
482
+ # Sort instances within each group
483
+ for instances in instances_by_doc_id.values():
484
+ instances.sort(key=lambda x: x.idx)
485
+ # iterate over different filters used
486
+ for filter_key in task.instances[0].filtered_resps.keys():
487
+ if not cli_args.process_with_media:
488
+ doc_iterator = create_iterator(enumerate(task.eval_docs_no_media), rank=RANK, limit=int(limit) if limit else None, world_size=WORLD_SIZE)
489
+ else:
490
+ doc_iterator = task.doc_iterator(rank=RANK, limit=limit, world_size=WORLD_SIZE)
491
+ doc_iterator_for_counting = itertools.islice(range(len(task.test_docs())), RANK, limit, WORLD_SIZE) if task.has_test_docs() else itertools.islice(range(len(task.validation_docs())), RANK, limit, WORLD_SIZE)
492
+ total_docs = sum(1 for _ in doc_iterator_for_counting)
493
+ pbar = tqdm(total=total_docs, desc=f"Postprocessing", disable=(RANK != 0))
494
+ for doc_id, doc in doc_iterator:
495
+ requests = instances_by_doc_id[doc_id]
496
+ metrics = task.process_results(doc, [req.filtered_resps[filter_key] for req in requests])
497
+ if log_samples:
498
+ target = task.doc_to_target(doc)
499
+ saved_doc = {}
500
+ for key, value in doc.items():
501
+ # If image is not in key
502
+ if "image" not in key:
503
+ # If audio is also not the value
504
+ if isinstance(value, dict) and "array" in value:
505
+ continue
506
+ else:
507
+ saved_doc[key] = value
508
+ filtered_arguments = []
509
+ for req in requests:
510
+ # check if req.args is a list of tuples, and each item in the list is a serializable object
511
+ for value in req.args:
512
+ if isinstance(value, (str, int, float, bool, list, dict, type(None))):
513
+ filtered_arguments.append(value)
514
+ # else:
515
+ # filtered_arguments.append(_handle_non_serializable(value))
516
+
517
+ example = {
518
+ "doc_id": doc_id,
519
+ "doc": saved_doc,
520
+ "target": target,
521
+ "arguments": filtered_arguments,
522
+ "resps": [req.resps for req in requests],
523
+ "filtered_resps": [req.filtered_resps[filter_key] for req in requests],
524
+ "doc_hash": hash_string(
525
+ json.dumps(
526
+ requests[0].doc,
527
+ indent=2,
528
+ default=handle_non_serializable,
529
+ ensure_ascii=False,
530
+ )
531
+ ),
532
+ "prompt_hash": hash_string(requests[0].arguments[0]),
533
+ "target_hash": hash_string(str(target)),
534
+ }
535
+ example.update(metrics)
536
+ task_output.logged_samples.append(example)
537
+ for metric, value in metrics.items():
538
+ task_output.sample_metrics[(metric, filter_key)].append(value)
539
+ pbar.update(1)
540
+
541
+ pbar.close()
542
+
543
+ if hasattr(lm, "_model"):
544
+ del lm._model
545
+ torch.cuda.empty_cache()
546
+
547
+ if WORLD_SIZE > 1:
548
+ # if multigpu, then gather data across all ranks to rank 0
549
+ # first gather logged samples across all ranks
550
+ for task_output in eval_tasks:
551
+ if log_samples:
552
+ # for task_name, task_samples in list(samples.items()):
553
+ full_samples = [None] * WORLD_SIZE if RANK == 0 else None
554
+ per_rank_samples = []
555
+ for sample in task_output.logged_samples:
556
+ per_rank_samples.append(sample)
557
+
558
+ torch.distributed.gather_object(
559
+ obj=per_rank_samples,
560
+ object_gather_list=full_samples,
561
+ dst=0,
562
+ )
563
+
564
+ if RANK == 0:
565
+ task_output.logged_samples = list(itertools.chain.from_iterable(full_samples))
566
+
567
+ # then collect metrics across all ranks
568
+ for metrics in task_output.sample_metrics:
569
+ metric_list = [None] * WORLD_SIZE if RANK == 0 else None
570
+ torch.distributed.gather_object(
571
+ obj=task_output.sample_metrics[metrics],
572
+ object_gather_list=metric_list,
573
+ dst=0,
574
+ )
575
+ if RANK == 0:
576
+ task_output.sample_metrics[metrics] = list(itertools.chain.from_iterable(metric_list))
577
+
578
+ dist.barrier() # Ensure all processes are synced before proceeding
579
+
580
+ if RANK == 0:
581
+ ### Aggregate results over all datapoints ###
582
+ # aggregate results ; run bootstrap CIs
583
+ for task_output in eval_tasks:
584
+ task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters)
585
+ (
586
+ results,
587
+ samples,
588
+ configs,
589
+ versions,
590
+ num_fewshot,
591
+ higher_is_better,
592
+ ) = consolidate_results(eval_tasks)
593
+
594
+ ### Calculate group metrics ###
595
+ if bool(results):
596
+ results, versions, show_group_table, *_ = consolidate_group_results(results, versions, task_dict)
597
+
598
+ results_agg, group_agg = prepare_print_tasks(task_dict, results)
599
+ subtask_list = get_subtask_list(task_dict)
600
+
601
+ # collect all higher_is_better values for metrics
602
+ # in the group's subtasks.
603
+ # TODO: clean this up ; unify with the below metric_list loop?
604
+ _higher_is_better = {}
605
+ for group, task_list in subtask_list.items():
606
+ if len(task_list) != 0: # subtask list will list "task_name": [] for solo tasks
607
+ for task in task_list:
608
+ for m, h in higher_is_better[task].items():
609
+ if m not in _higher_is_better.keys():
610
+ _higher_is_better[m] = h
611
+
612
+ if m in _higher_is_better and _higher_is_better[m] is not None and _higher_is_better[m] != h:
613
+ eval_logger.warning(f"Higher_is_better values for metric {m} in group {group} are not consistent. Defaulting to None.")
614
+ _higher_is_better[m] = None
615
+ higher_is_better[group] = _higher_is_better
616
+
617
+ results_dict = {
618
+ "results": dict(results_agg.items()),
619
+ **({"groups": dict(group_agg.items())} if (bool(group_agg) & show_group_table) else {}),
620
+ "group_subtasks": dict(reversed(subtask_list.items())),
621
+ "configs": dict(sorted(configs.items())),
622
+ "versions": dict(sorted(versions.items())),
623
+ "n-shot": dict(sorted(num_fewshot.items())),
624
+ "higher_is_better": dict(sorted(higher_is_better.items())),
625
+ "n-samples": {
626
+ task_output.task_name: {
627
+ "original": len(task_output.task.eval_docs),
628
+ "effective": min(
629
+ limit if limit else len(task_output.task.eval_docs),
630
+ len(task_output.task.eval_docs),
631
+ ),
632
+ }
633
+ for task_output in eval_tasks
634
+ },
635
+ }
636
+ if log_samples:
637
+ results_dict["samples"] = dict(samples)
638
+ else:
639
+ results_dict = None
640
+
641
+ if hasattr(lm, "accelerator"):
642
+ lm.accelerator.wait_for_everyone()
643
+
644
+ return results_dict
645
+
646
+
647
+ def request_caching_arg_to_dict(cache_requests: str) -> dict:
648
+ request_caching_args = {
649
+ "cache_requests": cache_requests in {"true", "refresh"},
650
+ "rewrite_requests_cache": cache_requests == "refresh",
651
+ "delete_requests_cache": cache_requests == "delete",
652
+ }
653
+
654
+ return request_caching_args
LaViDa/eval/lmms_eval/evaluator_utils.py ADDED
@@ -0,0 +1,488 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # credit to https://github.com/EleutherAI/lm-evaluation-harness
2
+ import collections
3
+ import inspect
4
+ import math
5
+ import pathlib
6
+ import sys
7
+ from typing import List, Optional, Tuple, Union
8
+
9
+ from lmms_eval.api.group import ConfigurableGroup
10
+ from lmms_eval.api.metrics import (
11
+ aggregate_subtask_metrics,
12
+ pooled_sample_stderr,
13
+ stderr_for_metric,
14
+ )
15
+ from lmms_eval.api.task import Task
16
+ from lmms_eval.utils import eval_logger, positional_deprecated
17
+
18
+
19
+ class TaskOutput:
20
+ """
21
+ Wrapper class for Task outputs.It contains various attributes and methods to manage and calculate metrics for the task.
22
+
23
+ Attributes:
24
+ task (object): The task object.
25
+ task_name (str): The name of the task.
26
+ task_config (dict): The configuration of the task.
27
+ version (str): The version of the task.
28
+ group_name (str): The name of the task group.
29
+ n_shot (int): The number of shots for the task.
30
+ task_alias (str): The alias of the task.
31
+ group_alias (str): The alias of the task group.
32
+ is_group (bool): Indicates if the task is a group.
33
+ logged_samples (list): The list of logged samples.
34
+ sample_len (int): The length of the samples.
35
+ sample_metrics (defaultdict): The dictionary of samples' metrics.
36
+ agg_metrics (defaultdict): The dictionary of aggregate metrics.
37
+
38
+ Methods:
39
+ from_taskdict(cls, task_name: str, task):
40
+ Creates a TaskOutput instance from a task dictionary.
41
+
42
+ calculate_aggregate_metric(bootstrap_iters=100000) -> None:
43
+ Calculates the aggregate metrics for the task.
44
+ """
45
+
46
+ def __init__(
47
+ self,
48
+ task=None,
49
+ task_name=None,
50
+ task_config=None,
51
+ version=None,
52
+ group_name=None,
53
+ n_shot=None,
54
+ task_alias=None,
55
+ group_alias=None,
56
+ is_group=None,
57
+ ):
58
+ self.task = task
59
+ self.task_config = task_config
60
+ self.task_name = task_name
61
+ self.group_name = group_name
62
+ self.version = version
63
+ self.n_shot = n_shot
64
+ self.task_alias = task_alias
65
+ self.group_alias = group_alias
66
+ self.is_group = is_group
67
+ self.logged_samples = []
68
+ self.sample_len = None
69
+ self.sample_metrics = collections.defaultdict(list)
70
+ self.agg_metrics = collections.defaultdict(list)
71
+ self.args = None
72
+
73
+ @classmethod
74
+ def from_taskdict(cls, task_name: str, task):
75
+ if isinstance(task, tuple):
76
+ group_name, task = task
77
+ else:
78
+ group_name = None
79
+ if not task:
80
+ # these gets filtered out in get_task_list
81
+ # once they are added to group hierarchy
82
+ is_group = True
83
+ return cls(task=task, task_name=task_name, is_group=is_group, group_name=group_name)
84
+ version = task.VERSION
85
+ task_config = dict(task.dump_config())
86
+ if (n_shot := task_config.get("num_fewshot")) == 0:
87
+ meta_config = task_config.get("metadata", {})
88
+ if isinstance(meta_config, dict):
89
+ n_shot = meta_config.get("num_fewshot", 0)
90
+ else:
91
+ eval_logger.info(f"No metadata found in task config for {task_name}, using default n_shot=0")
92
+ n_shot = 0
93
+ task_alias = task_config.get("alias")
94
+ group_alias = task_config.get("group_alias")
95
+ return cls(
96
+ task=task,
97
+ task_name=task_name,
98
+ task_config=task_config,
99
+ group_name=group_name,
100
+ version=version,
101
+ n_shot=n_shot,
102
+ task_alias=task_alias,
103
+ group_alias=group_alias,
104
+ )
105
+
106
+ def calculate_aggregate_metric(self, bootstrap_iters=100000) -> None:
107
+ for (metric, filter_key), items in self.sample_metrics.items():
108
+ if metric in self.task.aggregation():
109
+ agg_fn = self.task.aggregation()[metric]
110
+ metric_key = f"{metric},{filter_key}"
111
+ if "args" in inspect.signature(agg_fn).parameters:
112
+ self.agg_metrics[metric_key] = agg_fn(items, args=self.task.args)
113
+ else:
114
+ self.agg_metrics[metric_key] = agg_fn(items)
115
+ self.sample_len = len(items) # TODO: same sample size for each metric?
116
+ if isinstance(bootstrap_iters, int):
117
+ stderr_fn = stderr_for_metric(
118
+ metric=agg_fn,
119
+ bootstrap_iters=min(bootstrap_iters, 100) if metric in ["bleu", "chrf", "ter"] else bootstrap_iters,
120
+ )
121
+ self.agg_metrics[f"{metric}_stderr,{filter_key}"] = stderr_fn(items) if (stderr_fn and len(items) > 1) else "N/A"
122
+ else:
123
+ raise ValueError(f"Received bootstrap_iters '{bootstrap_iters}' but expected an integer. Set to 0 to turn off stderr calculations.")
124
+
125
+ def __repr__(self):
126
+ return f"TaskOutput(task_name={self.task_name}, " f"group_name={self.group_name}, " f"version={self.version}, " f"n_shot={self.n_shot}, " f"task_alias={self.task_alias}, " f"group_alias={self.group_alias})"
127
+
128
+
129
+ def get_task_list(task_dict: dict) -> List[TaskOutput]:
130
+ outputs = []
131
+ for task_name, task_obj in task_dict.items():
132
+ if isinstance(task_obj, dict):
133
+ _outputs = get_task_list(task_obj)
134
+ outputs.extend(_outputs)
135
+ else:
136
+ task_output = TaskOutput.from_taskdict(task_name, task_obj)
137
+ outputs.append(task_output)
138
+
139
+ return outputs
140
+
141
+
142
+ def get_subtask_list(task_dict, task_root=None, depth=0):
143
+ subtask_list = {}
144
+ for group_obj, task_obj in task_dict.items():
145
+ if isinstance(group_obj, ConfigurableGroup):
146
+ # group_name = group_obj.group_name
147
+ group_name = group_obj.group_name
148
+ else:
149
+ group_name = group_obj
150
+ if isinstance(task_obj, dict):
151
+ _subtask_list = get_subtask_list(task_obj, task_root=group_name, depth=depth + 1)
152
+ if task_root:
153
+ subtask_list.setdefault((task_root, depth), []).extend([_task for (_task, _depth) in _subtask_list.keys() if (_depth - 1) == depth])
154
+
155
+ subtask_list = {**subtask_list, **_subtask_list}
156
+ else:
157
+ if isinstance(task_obj, ConfigurableGroup):
158
+ # group_or_task_name = task_obj.group_name
159
+ group_or_task_name = task_obj.group_name
160
+ elif isinstance(task_obj, Task):
161
+ # group_or_task_name = task_obj.task_name
162
+ group_or_task_name = task_obj.task_name
163
+
164
+ if task_root is None:
165
+ subtask_list.setdefault((group_or_task_name, depth), [])
166
+ else:
167
+ subtask_list.setdefault((task_root, depth), []).append(group_or_task_name)
168
+
169
+ if depth == 0:
170
+ _subtask_list = {}
171
+ for group_key, task_list in subtask_list.items():
172
+ group_name, depth = group_key
173
+ _subtask_list[group_name] = task_list
174
+ subtask_list = _subtask_list
175
+
176
+ return subtask_list
177
+
178
+
179
+ def print_writeout(task) -> None:
180
+ for inst in task.instances:
181
+ # print the prompt for the first few documents
182
+ if inst.doc_id < 1:
183
+ eval_logger.info(
184
+ f"Task: {task}; document {inst.doc_id}; context prompt (starting on next line):\
185
+ \n{inst.args[0]}\n(end of prompt on previous line)\ntarget string or answer choice index (starting on next line):\n{task.doc_to_target(inst.doc)}\n(end of target on previous line)"
186
+ )
187
+ eval_logger.info(f"Request: {str(inst)}")
188
+
189
+
190
+ def get_sample_size(task, limit: Optional[int]) -> Union[int, None]:
191
+ if limit is not None:
192
+ limit = int(math.ceil(len(task.eval_docs) * limit)) if limit < 1.0 else int(limit)
193
+ return limit
194
+
195
+
196
+ def prepare_print_tasks(
197
+ task_dict: dict,
198
+ results: dict,
199
+ task_depth=0,
200
+ group_depth=0,
201
+ ) -> Tuple[dict, dict]:
202
+ """
203
+ @param task_dict: Dictionary representing the group hierarchy of tasks. Each key is a group name and its
204
+ value is a list of task names.
205
+ @param results: Dictionary containing the results of each task. Each key is a
206
+ group name and its value is a dictionary of task results.
207
+ @param task_depth: The indentation level for printing the task
208
+ hierarchy. Default is 0.
209
+ @param group_depth: The indentation level for printing the group
210
+ hierarchy. Default is 0.
211
+ @return: A tuple of two dictionaries: results_agg and groups_agg. results_agg contains
212
+ aggregated results for each task, and groups_agg contains aggregated results for each group.
213
+
214
+ Prepares the task hierarchy and aggregates the results for each task and group recursively for printing.
215
+ """
216
+
217
+ def _sort_task_dict(task_dict):
218
+ """
219
+ Helper utility. Sorts the task dict at the current level of the hierarchy based on alphabetized task name.
220
+ Required so that we end up sorting within each sub-header correctly.
221
+ """
222
+
223
+ return dict(
224
+ sorted(
225
+ task_dict.items(),
226
+ key=lambda item: item[0].group_name if isinstance(item[0], ConfigurableGroup) else item[0],
227
+ )
228
+ )
229
+
230
+ task_agg = collections.defaultdict(dict)
231
+ group_agg = collections.defaultdict(dict)
232
+ task_dict = _sort_task_dict(task_dict)
233
+ for task_or_group_name, task_or_group_obj in task_dict.items():
234
+ tab_string = " " * task_depth + "- " if task_depth > 0 else ""
235
+ if isinstance(task_or_group_name, ConfigurableGroup):
236
+ # string_name = task_or_group_name.group_name
237
+ name = task_or_group_name.group_name
238
+ from_configurable_group = True
239
+ task_or_group_obj = _sort_task_dict(task_or_group_obj)
240
+ elif isinstance(task_or_group_name, str):
241
+ name = task_or_group_name
242
+ if isinstance(task_or_group_obj, Task):
243
+ # string_name = task_or_group_obj.task_name
244
+ name = task_or_group_obj.task_name
245
+ from_configurable_group = False
246
+
247
+ task_agg[name] = results[name].copy()
248
+ if from_configurable_group:
249
+ if task_or_group_name.group_alias is not None:
250
+ alias = task_or_group_name.group_alias
251
+ else:
252
+ alias = task_or_group_name.group
253
+ else:
254
+ if "alias" in task_agg[name]:
255
+ alias = task_agg[name]["alias"]
256
+ else:
257
+ alias = name
258
+
259
+ task_agg[name]["alias"] = tab_string + alias
260
+ if "samples" in task_agg[name]:
261
+ task_agg[name].pop("samples")
262
+
263
+ if from_configurable_group and (" " not in results[name]):
264
+ group_tab_string = " " * group_depth + "- " if group_depth > 0 else ""
265
+ group_agg[name] = results[name].copy()
266
+ group_agg[name]["alias"] = group_tab_string + alias
267
+ if "samples" in group_agg[name]:
268
+ group_agg[name].pop("samples")
269
+
270
+ if isinstance(task_or_group_obj, dict):
271
+ task_depth += 1
272
+ group_depth += 1
273
+ _task_agg, _group_agg = prepare_print_tasks(task_or_group_obj, results, task_depth, group_depth)
274
+ task_agg = {
275
+ **task_agg,
276
+ **_task_agg,
277
+ }
278
+ group_agg = {**group_agg, **_group_agg}
279
+ task_depth -= 1
280
+ group_depth -= 1
281
+ return task_agg, group_agg
282
+
283
+
284
+ def consolidate_results(
285
+ eval_tasks: List[TaskOutput],
286
+ ) -> Tuple[dict, dict, dict, dict, dict, dict]:
287
+ """
288
+ @param eval_tasks: list(TaskOutput).
289
+ @return: A tuple containing the consolidated results, samples, configs, versions, and num_fewshot.
290
+
291
+ Consolidates the results of multiple evaluation tasks into a single structure.
292
+
293
+ The method iterates over each evaluation instance and extracts relevant information to create the consolidated
294
+ results structure. The consolidated results structure has the following properties:
295
+
296
+ - results: A defaultdict with task names as keys and dictionaries as values. Each dictionary contains
297
+ metric/filter pairs as keys and corresponding metric values as values. The "alias" key is used to store task
298
+ aliases specified in the task configuration.
299
+ - samples: A defaultdict with task names as keys and lists of log samples as values.
300
+ - configs: A defaultdict with task names as keys and task configurations as values.
301
+ - versions: A defaultdict with task names as keys and task versions as values.
302
+ - num_fewshot: A defaultdict with task names as keys and number of few-shot samples as values.
303
+ - higher_is_better: A defaultdict with task names as keys and indicators of whether higher values are better
304
+ for each metric as values.
305
+
306
+ The method then returns the consolidated results, samples, configs, versions, and num_fewshot as a tuple.
307
+ """
308
+ # stores the final result for each task, for each metric/filter pair.
309
+ results = collections.defaultdict(dict)
310
+ # logs info about each document evaluated.
311
+ samples = collections.defaultdict(list)
312
+ # store num-fewshot value per task
313
+ num_fewshot = collections.defaultdict(int)
314
+ # Tracks the YAML configs of all chosen task
315
+ configs = collections.defaultdict(dict)
316
+ # Tracks each task's version.
317
+ versions = collections.defaultdict(dict)
318
+ # Track `higher_is_better` for each metric
319
+ higher_is_better = collections.defaultdict(dict)
320
+
321
+ for task_output in eval_tasks:
322
+ if "task_alias" in (task_config := task_output.task_config):
323
+ results[task_output.task_name]["alias"] = task_config["task_alias"]
324
+ else:
325
+ results[task_output.task_name]["alias"] = task_output.task_name
326
+ if group_alias := task_output.group_alias:
327
+ if group_alias not in results and (group_name := task_output.group_name):
328
+ results[group_name]["alias"] = group_alias
329
+ num_fewshot[task_output.task_name] = task_output.n_shot
330
+ configs[task_output.task_name] = task_output.task_config
331
+ versions[task_output.task_name] = task_output.version
332
+ samples[task_output.task_name] = task_output.logged_samples
333
+ higher_is_better[task_output.task_name] = task_output.task.higher_is_better()
334
+ for (metric, filter_key), items in task_output.sample_metrics.items():
335
+ metric_key = f"{metric},{filter_key}"
336
+ results[task_output.task_name][metric_key] = task_output.agg_metrics[metric_key]
337
+ results[task_output.task_name]["samples"] = task_output.sample_len
338
+ results[task_output.task_name][f"{metric}_stderr,{filter_key}"] = task_output.agg_metrics[f"{metric}_stderr,{filter_key}"]
339
+ return results, samples, configs, versions, num_fewshot, higher_is_better
340
+
341
+
342
+ def consolidate_group_results(
343
+ results,
344
+ versions,
345
+ task_dict,
346
+ task_root=None,
347
+ show_group_table=False,
348
+ task_aggregation_list=None,
349
+ ) -> Tuple[dict, dict, bool, Union[None, dict]]:
350
+ """
351
+ (Recursively) calculates groups' aggregated metrics and updates the results and versions dictionaries with this info.
352
+
353
+ @return: a tuple [results, versions, show_group_table, task_aggregation_list] with formats described below:
354
+
355
+ - results: A defaultdict with task names (and, after this function is called, group names of
356
+ groups that perform aggregation) as keys, and dictionaries with "alias" and metric,filter_name pairs as keys.
357
+ - versions: A defaultdict with task names (and, after this function is called, group names of
358
+ groups that perform aggregation) as keys, and float values representing the task or group's version if a version is specified. (defaulting to None).
359
+ - show_group_table: a boolean which is true if there exists a group that requires printing of its aggregated scores in a group table.
360
+ - task_aggregation_list: a defaultdict listing the subtasks to average over to produce a given group's end metric.
361
+
362
+ The method then returns the updated results, versions, show_group_table, and task_aggregation_list as a tuple.
363
+ In the top-level invocation of this function, task_aggregation_list is ignored.
364
+ """
365
+ if task_root is None:
366
+ task_root = {}
367
+
368
+ if task_aggregation_list is None:
369
+ task_aggregation_list = {}
370
+
371
+ for group_or_task, group_or_task_info in task_dict.items():
372
+ # Convert to string
373
+ if isinstance(group_or_task, ConfigurableGroup):
374
+ group_config = group_or_task.config
375
+ group_or_task = group_or_task.group_name
376
+ else:
377
+ group_config = None
378
+
379
+ if isinstance(group_or_task_info, Task):
380
+ if task_root:
381
+ task_aggregation_list.setdefault(task_root, []).append(group_or_task_info.task_name)
382
+ else:
383
+ (
384
+ results,
385
+ versions,
386
+ show_group_table,
387
+ _task_aggregation_list,
388
+ ) = consolidate_group_results(
389
+ results,
390
+ versions,
391
+ group_or_task_info,
392
+ group_or_task,
393
+ show_group_table,
394
+ task_aggregation_list,
395
+ )
396
+ if task_root:
397
+ task_aggregation_list.setdefault(task_root, []).extend(task_aggregation_list.get(group_or_task, []))
398
+
399
+ if (group_config is None) or (group_config["aggregate_metric_list"] is None):
400
+ results[group_or_task][" "] = " "
401
+ continue
402
+
403
+ if "aggregate_metric_list" in group_config:
404
+ agg_metric_list = group_config["aggregate_metric_list"]
405
+
406
+ show_group_table = show_group_table | bool(group_config["aggregate_metric_list"])
407
+
408
+ task_list = _task_aggregation_list[group_or_task]
409
+
410
+ metric_list = list({key for task in task_list for key in results[task].keys() if "_stderr" not in key and key not in ["task", "alias", "samples"]})
411
+ for metric in metric_list:
412
+ stderr = "_stderr,".join(metric.split(","))
413
+
414
+ # gather metrics, sizes, and stderrs from subtasks
415
+ metrics = [results[task][metric] for task in task_list if metric in results[task]] # TODO: copy?
416
+ stderrs = [results[task][stderr] for task in task_list if stderr in results[task]]
417
+ sizes = [results[task]["samples"] for task in task_list if metric in results[task]]
418
+
419
+ for metric_config in agg_metric_list:
420
+ for filter_name in metric_config["filter_list"]:
421
+ # if metric != ",".join([metric_config["metric"], filter_name]):
422
+ # continue
423
+ if metric_config["metric"] not in metric:
424
+ continue
425
+
426
+ # compute group's pooled metric and stderr
427
+ if metric_config["aggregation"] == "mean":
428
+ aggregate_fn = aggregate_subtask_metrics
429
+ elif callable(metric_config["aggregation"]):
430
+ aggregate_fn = metric_config["aggregation"]
431
+ else:
432
+ raise ValueError(f"Currently, only 'mean' is supported for automatically aggregating scores across groups' subtasks. Got '{metric_config['aggregation']}' for group '{group_or_task}'")
433
+
434
+ results[group_or_task][metric] = aggregate_fn(
435
+ metrics,
436
+ sizes,
437
+ metric_config["weight_by_size"],
438
+ )
439
+ # TODO: calculate groups' metrics using arbitrary agg fns
440
+ if "N/A" in stderrs:
441
+ results[group_or_task][stderr] = "N/A"
442
+ else:
443
+ # NOTE: this assumes we are using the mean to aggregate. There are warnings about this elsewhere
444
+ results[group_or_task][stderr] = pooled_sample_stderr(stderrs, sizes)
445
+
446
+ results[group_or_task]["samples"] = sum(sizes)
447
+ group_metadata = group_config.get("metadata", None)
448
+ if group_metadata is not None:
449
+ versions[group_or_task] = group_metadata.get("version", None)
450
+ # print(results)
451
+ return results, versions, show_group_table, task_aggregation_list
452
+
453
+
454
+ @positional_deprecated
455
+ def find_test_root(start_path: pathlib.Path) -> pathlib.Path:
456
+ """
457
+ Search upward in the directory tree to a maximum of three layers
458
+ to find and return the package root (containing the 'tests' folder)
459
+ """
460
+ cur_path = start_path.resolve()
461
+ max_layers = 3
462
+ for _ in range(max_layers):
463
+ if (cur_path / "tests" / "test_version_stable.py").exists():
464
+ return cur_path
465
+ else:
466
+ cur_path = cur_path.parent.resolve()
467
+ raise FileNotFoundError(f"Unable to find package root within {max_layers} upwards" + f"of {start_path}")
468
+
469
+
470
+ @positional_deprecated
471
+ def run_task_tests(task_list: List[str]):
472
+ """
473
+ Find the package root and run the tests for the given tasks
474
+ """
475
+ import pytest
476
+
477
+ package_root = find_test_root(start_path=pathlib.Path(__file__))
478
+ task_string = " or ".join(task_list)
479
+ args = [
480
+ f"{package_root}/tests/test_version_stable.py",
481
+ f"--rootdir={package_root}",
482
+ "-k",
483
+ f"{task_string}",
484
+ ]
485
+ sys.path.append(str(package_root))
486
+ pytest_return_val = pytest.main(args)
487
+ if pytest_return_val:
488
+ raise ValueError(f"Not all tests for the specified tasks ({task_list}) ran successfully! Error code: {pytest_return_val}")
LaViDa/eval/lmms_eval/filters/__init__.py ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from lmms_eval.api.filter import Filter, FilterEnsemble
2
+
3
+ from . import extraction, selection, transformation
4
+
5
+ FILTER_REGISTRY = {
6
+ "take_first": selection.TakeFirstFilter,
7
+ "regex": extraction.RegexFilter,
8
+ "majority_vote": selection.MajorityVoteFilter,
9
+ "take_first_k": selection.TakeKFilter,
10
+ "remove_whitespace": extraction.WhitespaceFilter,
11
+ "lowercase": transformation.LowercaseFilter,
12
+ "uppercase": transformation.UppercaseFilter,
13
+ "map": transformation.MapFilter,
14
+ "multi_choice_regex": extraction.MultiChoiceRegexFilter,
15
+ # TODO: implement this filter. either it should take in an arbitrary "scoring"/reward function
16
+ # that takes an input and returns a scalar and then should select the max reward,
17
+ # or should implement different filters for different ways of handling a reward model's inference.
18
+ # "arg_max": selection.ArgMaxFilter,
19
+ }
20
+
21
+
22
+ def get_filter(filter_name):
23
+ if filter_name in FILTER_REGISTRY:
24
+ return FILTER_REGISTRY[filter_name]
25
+ else:
26
+ return filter_name
27
+
28
+
29
+ def build_filter_ensemble(filter_name, components):
30
+ """
31
+ Create a filtering pipeline.
32
+ """
33
+ filters = []
34
+ for function, kwargs in components:
35
+ if kwargs is None:
36
+ f = get_filter(function)()
37
+ else:
38
+ # create a filter given its name in the registry
39
+ f = get_filter(function)(**kwargs) # TODO: pass kwargs to filters properly
40
+ # add the filter as a pipeline step
41
+ filters.append(f)
42
+
43
+ return FilterEnsemble(name=filter_name, filters=filters)
LaViDa/eval/lmms_eval/filters/decontamination.py ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from lmms_eval.api.filter import Filter
2
+
3
+
4
+ class DecontaminationFilter(Filter):
5
+ """
6
+ A filter which evaluates
7
+ """
8
+
9
+ name = "track_decontamination"
10
+
11
+ def __init__(self, path) -> None:
12
+ """
13
+
14
+ TODO: make sure only ever run one time on the train set (should this be cached as a class var? keyed by value for "path").
15
+ should further cache result on a given (task_name, doc_id)
16
+ """
17
+ self._decontam_results = None
18
+
19
+ def apply(self, resps, docs) -> None:
20
+ """
21
+ Return {"no_contamination", "only_contamination"} keys for the 2 different subsets
22
+ """
23
+ pass
LaViDa/eval/lmms_eval/filters/extraction.py ADDED
@@ -0,0 +1,282 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import re
3
+ import sys
4
+ import unicodedata
5
+
6
+ import openai
7
+
8
+ from lmms_eval.api.filter import Filter
9
+
10
+
11
+ class WhitespaceFilter(Filter):
12
+ """ """
13
+
14
+ def __init__(self) -> None:
15
+ pass
16
+
17
+ def apply(self, resps, docs):
18
+ def filter_set(inst):
19
+ filtered_resp = []
20
+ for resp in inst:
21
+ if resp.startswith(" "):
22
+ resp = resp[1:]
23
+
24
+ filtered_resp.append(resp)
25
+
26
+ return filtered_resp
27
+
28
+ filtered_resps = [filter_set(resp) for resp in resps]
29
+
30
+ return filtered_resps
31
+
32
+
33
+ class RegexFilter(Filter):
34
+ """ """
35
+
36
+ def __init__(
37
+ self,
38
+ regex_pattern: str = r"#### (\-?[0-9\.\,]+)",
39
+ group_select=0,
40
+ fallback: str = "[invalid]",
41
+ ) -> None:
42
+ """
43
+ pass a string `regex` to run `re.compile(r"regex")` on.
44
+ `fallback` defines the output returned if no matches for the regex are located.
45
+ """
46
+ self.regex_pattern = regex_pattern
47
+ self.regex = re.compile(regex_pattern)
48
+ self.group_select = group_select
49
+ self.fallback = fallback
50
+
51
+ def apply(self, resps, docs):
52
+ # here, we assume we have a list, in which each element is
53
+ # a list of model responses for some particular input/target pair.
54
+ # so we process each of these (same input/target response sets)
55
+ # independently (and keep them a list.)
56
+ def filter_set(inst):
57
+ filtered = []
58
+ for resp in inst:
59
+ match = self.regex.findall(resp)
60
+ if match:
61
+ match = match[self.group_select]
62
+ if isinstance(match, tuple):
63
+ match = [m for m in match if m][0]
64
+ match = match.strip()
65
+ else:
66
+ match = self.fallback
67
+ filtered.append(match)
68
+ return filtered
69
+
70
+ # print(resps)
71
+ filtered_resps = list(map(lambda x: filter_set(x), resps))
72
+ # print(filtered_resps)
73
+
74
+ return filtered_resps
75
+
76
+
77
+ class MultiChoiceRegexFilter(RegexFilter):
78
+ """
79
+ A filter used to extract a model's answer on multiple choice questions with
80
+ letter answers. assumes each document has a "choices" field
81
+ containing the list of answer choices and that the answer label symbols
82
+ are of the form (A), (B), (C), ... or A, B, C.
83
+ """
84
+
85
+ def __init__(
86
+ self,
87
+ regex_pattern: str = r"#### (\-?[0-9\.\,]+)",
88
+ group_select=0,
89
+ fallback: str = "[invalid]",
90
+ ignore_case=False,
91
+ ignore_punctuation=False,
92
+ regexes_to_ignore=None,
93
+ ) -> None:
94
+ """
95
+ regex_pattern: The basic regex pattern to use. If fails to match, we will use the customized match procedure
96
+ - step 1 : We parse the choices between ([A-Z])s then try to find these choices in the response.
97
+ - step 2 : We parse the choice with regex :[\s]*([A-?]), where ? varies by number of choices.
98
+ group_select: Selects the (group_select)th match from the findall result.
99
+ ignore_case: Ignores the case during step 1 matching
100
+ ignore_punctuation: Remove the punctuation during step 1 matching
101
+ regexes_to_ignore: Remove these regexes during step 1 matching
102
+ """
103
+ super().__init__(regex_pattern, group_select, fallback)
104
+ self.ignore_case = ignore_case
105
+ self.ignore_punctuation = ignore_punctuation
106
+ self.regexes_to_ignore = regexes_to_ignore
107
+
108
+ def apply(self, resps, docs):
109
+ # here, we assume we have a list, in which each element is
110
+ # a list of model responses for some particular input/target pair.
111
+ # so we process each of these (same input/target response sets)
112
+ # independently (and keep them a list.)
113
+
114
+ def find_match(regex, resp, convert_dict={}):
115
+ match = regex.findall(resp)
116
+ if match:
117
+ match = match[self.group_select]
118
+ if isinstance(match, tuple):
119
+ match = [m for m in match if m][0]
120
+ match = match.strip()
121
+ if match and match in convert_dict:
122
+ match = convert_dict[match]
123
+ return match
124
+
125
+ punct_tbl = dict.fromkeys(i for i in range(sys.maxunicode) if unicodedata.category(chr(i)).startswith("P"))
126
+
127
+ def filter_ignores(st):
128
+ if self.regexes_to_ignore is not None:
129
+ for s in self.regexes_to_ignore:
130
+ st = re.sub(s, "", st)
131
+
132
+ if self.ignore_case:
133
+ st = st.lower()
134
+
135
+ if self.ignore_punctuation:
136
+ # https://stackoverflow.com/a/266162
137
+ st = st.translate(punct_tbl)
138
+ return st
139
+
140
+ filtered_resps = []
141
+
142
+ for r, doc in zip(resps, docs):
143
+ fallback_regexes = []
144
+ choice_to_alpha = {}
145
+ next_alpha = "A"
146
+
147
+ without_paren_fallback_regexes = []
148
+ without_paren_to_target = {}
149
+
150
+ choices = doc["choices"]
151
+ for c in choices:
152
+ m = filter_ignores(c.strip())
153
+ fallback_regexes.append(f"{re.escape(m)}")
154
+ choice_to_alpha[m] = f"({next_alpha})"
155
+
156
+ without_paren_fallback_regexes.append(next_alpha)
157
+ without_paren_to_target[next_alpha] = f"({next_alpha})"
158
+
159
+ next_alpha = chr(ord(next_alpha) + 1)
160
+ fallback_regex = re.compile("|".join(fallback_regexes))
161
+ without_paren_fallback_regex = "|".join(without_paren_fallback_regexes)
162
+ without_paren_fallback_regex = re.compile(f":[\s]*({without_paren_fallback_regex})")
163
+
164
+ filtered = []
165
+ for resp in r:
166
+ match = find_match(self.regex, resp)
167
+ if not match:
168
+ match = find_match(fallback_regex, filter_ignores(resp), choice_to_alpha)
169
+ if not match:
170
+ match = find_match(without_paren_fallback_regex, resp, without_paren_to_target)
171
+ if not match:
172
+ match = self.fallback
173
+ filtered.append(match)
174
+ filtered_resps.append(filtered)
175
+
176
+ return filtered_resps
177
+
178
+
179
+ class ExtendedRegexFilter(RegexFilter):
180
+ punct_tbl = dict.fromkeys(i for i in range(sys.maxunicode) if unicodedata.category(chr(i)).startswith("P"))
181
+
182
+ def __init__(
183
+ self,
184
+ regex_pattern: str = r"#### (\-?[0-9\.\,]+)",
185
+ group_select=0,
186
+ fallback: str = "[invalid]",
187
+ ignore_case=False,
188
+ ignore_punctuation=False,
189
+ regexes_to_ignore=None,
190
+ ) -> None:
191
+ super().__init__(regex_pattern, group_select, fallback)
192
+ self.ignore_case = ignore_case
193
+ self.ignore_punctuation = ignore_punctuation
194
+ self.regexes_to_ignore = regexes_to_ignore
195
+
196
+ def filter_ignores(self, st):
197
+ if self.regexes_to_ignore is not None:
198
+ for s in self.regexes_to_ignore:
199
+ st = re.sub(s, "", st)
200
+
201
+ if self.ignore_case:
202
+ st = st.lower()
203
+
204
+ if self.ignore_punctuation:
205
+ # https://stackoverflow.com/a/266162
206
+ st = st.translate(self.punct_tbl)
207
+ return st
208
+
209
+ def find_match(self, regex, resp, convert_dict={}):
210
+ match = regex.findall(resp)
211
+ if match:
212
+ match = match[self.group_select]
213
+ if isinstance(match, tuple):
214
+ match = [m for m in match if m][0]
215
+ match = match.strip()
216
+ if match and match in convert_dict:
217
+ match = convert_dict[match]
218
+ return match
219
+
220
+
221
+ # Designed for the AI2D/RealworldQA dataset
222
+ class SimpleMultiChoiceRegexFilter(ExtendedRegexFilter):
223
+ def __init__(self, *args, **kwargs):
224
+ """
225
+ regex_pattern: The basic regex pattern to use. If fails to match, we will use the customized match procedure
226
+ - step 1 : We parse the choices between ([A-Z])s then try to find these choices in the response.
227
+ - step 2 : We parse the choice with regex :[\s]*([A-?]), where ? varies by number of choices.
228
+ group_select: Selects the (group_select)th match from the findall result.
229
+ ignore_case: Ignores the case during step 1 matching
230
+ ignore_punctuation: Remove the punctuation during step 1 matching
231
+ regexes_to_ignore: Remove these regexes during step 1 matching
232
+ """
233
+ super().__init__(*args, **kwargs)
234
+
235
+ def apply(self, resps, docs):
236
+ # here, we assume we have a list, in which each element is
237
+ # a list of model responses for some particular input/target pair.
238
+ # so we process each of these (same input/target response sets)
239
+ # independently (and keep them a list.)
240
+
241
+ filtered_resps = []
242
+
243
+ for r, doc in zip(resps, docs):
244
+ fallback_regexes = []
245
+ choice_to_alpha = {}
246
+ next_alpha = "A"
247
+
248
+ without_paren_fallback_regexes = []
249
+ without_paren_to_target = {}
250
+
251
+ # Regex to extract multiple choice options from the question
252
+ multiple_choices_regex = re.compile(r"\b([A-Z])\.\s+([^\n]*)")
253
+ matches = multiple_choices_regex.findall(doc["question"])
254
+
255
+ # Build regex patterns and mappings for each choice
256
+ for m in matches:
257
+ choice_text = m[1].strip()
258
+ fallback_regexes.append(f"{re.escape(choice_text)}")
259
+ choice_to_alpha[choice_text] = next_alpha
260
+
261
+ next_alpha = chr(ord(next_alpha) + 1)
262
+
263
+ # Compile regex to match any of the extracted choices
264
+ fallback_regex = re.compile("|".join(fallback_regexes))
265
+
266
+ # Process each response
267
+ filtered = []
268
+ for resp in r:
269
+ # Remove any punctuation and extra spaces
270
+ cleaned_resp = re.sub(r"[^\w\s]", "", resp).strip()
271
+ # Try to match cleaned response with the choice text
272
+ match = fallback_regex.search(cleaned_resp)
273
+ if match and match.group() in choice_to_alpha:
274
+ # Map the matched choice text back to its corresponding letter
275
+ filtered.append(choice_to_alpha[match.group()])
276
+ else:
277
+ # If no match, return the cleaned response
278
+ filtered.append(cleaned_resp)
279
+
280
+ filtered_resps.append(filtered[0])
281
+
282
+ return filtered_resps
LaViDa/eval/lmms_eval/filters/selection.py ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from collections import Counter
2
+
3
+ from lmms_eval.api.filter import Filter
4
+ from lmms_eval.api.registry import register_filter
5
+
6
+ # TODO: implement "arg_max" filter. either it should take in an arbitrary "scoring"/reward function
7
+ # that takes an input and returns a scalar and then should select the max reward,
8
+ # or should implement different filters for different ways of handling a reward model's inference.
9
+
10
+
11
+ @register_filter("take_first")
12
+ class TakeFirstFilter(Filter):
13
+ def __init__(self) -> None:
14
+ """
15
+ Can define custom behavior here, if an individual instantiation of a Filter class should have state.
16
+ """
17
+
18
+ def apply(self, resps, docs):
19
+ """
20
+ Assuming each entry of `resps` is a list of model responses, we discard all but the first response.
21
+ """
22
+ return map(lambda r: r[0], resps)
23
+
24
+
25
+ @register_filter("take_first_k")
26
+ class TakeKFilter(Filter):
27
+ def __init__(self, **kwargs) -> None:
28
+ self.k = kwargs.pop("k")
29
+
30
+ super().__init__(**kwargs)
31
+
32
+ def apply(self, resps, docs):
33
+ # need resp to be subscriptable to check below
34
+ resps = list(resps)
35
+ # check we have at least k responses per doc, else we can't take the first k
36
+ assert len(resps[0]) >= self.k, f"Need at least {self.k} responses per doc to take first {self.k}, but got {len(resps[0])} only! Please increase TaskConfig.repeats ."
37
+ selected = map(lambda r: r[: self.k], resps)
38
+ return selected
39
+
40
+
41
+ @register_filter("majority_vote")
42
+ class MajorityVoteFilter(Filter):
43
+ def __init__(self) -> None:
44
+ """
45
+ Can define custom behavior here, if an individual instantiation of a Filter class should have state.
46
+ """
47
+
48
+ def apply(self, resps, docs):
49
+ """
50
+ Each entry of `resps` is a list of model responses.
51
+ We select the response that occurs most frequently in each entry of `resps`.
52
+ """
53
+
54
+ def select_majority(resp):
55
+ counts = Counter(resp)
56
+ vote = counts.most_common(1)[0][0]
57
+ return vote
58
+
59
+ return map(lambda r: [select_majority(r)], resps)
LaViDa/eval/lmms_eval/filters/transformation.py ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from lmms_eval.api.filter import Filter
2
+
3
+
4
+ class LowercaseFilter(Filter):
5
+ def __init__(self) -> None:
6
+ pass
7
+
8
+ def apply(self, resps, docs):
9
+ def filter_set(inst):
10
+ return [resp.lower() for resp in inst]
11
+
12
+ return [filter_set(resp) for resp in resps]
13
+
14
+
15
+ class UppercaseFilter(Filter):
16
+ def __init__(self) -> None:
17
+ pass
18
+
19
+ def apply(self, resps, docs):
20
+ def filter_set(inst):
21
+ return [resp.upper() for resp in inst]
22
+
23
+ return [filter_set(resp) for resp in resps]
24
+
25
+
26
+ class MapFilter(Filter):
27
+ def __init__(self, mapping_dict: dict = {}, default_value=None) -> None:
28
+ """
29
+ Initializes the MapFilter with a given mapping dictionary and default value.
30
+
31
+ Args:
32
+ - mapping_dict (dict): A dictionary containing the key-value mappings.
33
+ Default is an empty dictionary.
34
+ - default_value (Any): The value to be returned when a key is not found in the mapping_dict.
35
+ Default is None.
36
+
37
+ Example:
38
+ mapper = MapFilter({'A': 1, 'B': 2}, default_value=0)
39
+ """
40
+ assert isinstance(mapping_dict, dict), "Provided mapping_dict is not a dictionary"
41
+ self.mapping_dict = mapping_dict
42
+ self.default_value = default_value
43
+
44
+ def apply(self, resps, docs):
45
+ def filter_set(inst):
46
+ return [self.mapping_dict.get(resp, self.default_value) for resp in inst]
47
+
48
+ return [filter_set(resp) for resp in resps]
LaViDa/eval/lmms_eval/loggers/__init__.py ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ from .evaluation_tracker import EvaluationTracker
2
+ from .wandb_logger import WandbLogger
LaViDa/eval/lmms_eval/loggers/evaluation_tracker.py ADDED
@@ -0,0 +1,452 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # code is adapted from https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/loggers/evaluation_tracker.py
2
+ import json
3
+ import os
4
+ import re
5
+ import time
6
+ from collections import defaultdict
7
+ from dataclasses import asdict, dataclass
8
+ from datetime import datetime
9
+ from pathlib import Path
10
+
11
+ from datasets import load_dataset
12
+ from datasets.utils.metadata import MetadataConfigs
13
+ from huggingface_hub import DatasetCard, DatasetCardData, HfApi, hf_hub_url
14
+ from huggingface_hub.utils import build_hf_headers, get_session, hf_raise_for_status
15
+
16
+ from lmms_eval.utils import (
17
+ eval_logger,
18
+ get_datetime_str,
19
+ get_file_datetime,
20
+ get_file_task_name,
21
+ get_results_filenames,
22
+ get_sample_results_filenames,
23
+ handle_non_serializable,
24
+ hash_string,
25
+ sanitize_list,
26
+ sanitize_model_name,
27
+ sanitize_task_name,
28
+ )
29
+
30
+
31
+ @dataclass(init=False)
32
+ class GeneralConfigTracker:
33
+ """
34
+ Tracker for the evaluation parameters.
35
+
36
+ Attributes:
37
+ model_source (str): Source of the model (e.g. Hugging Face, GGUF, etc.)
38
+ model_name (str): Name of the model.
39
+ model_name_sanitized (str): Sanitized model name for directory creation.
40
+ start_time (float): Start time of the experiment. Logged at class init.
41
+ end_time (float): Start time of the experiment. Logged when calling [`GeneralConfigTracker.log_end_time`]
42
+ total_evaluation_time_seconds (str): Inferred total evaluation time in seconds (from the start and end times).
43
+ """
44
+
45
+ model_source: str = None
46
+ model_name: str = None
47
+ model_name_sanitized: str = None
48
+ system_instruction: str = None
49
+ system_instruction_sha: str = None
50
+ fewshot_as_multiturn: bool = None
51
+ chat_template: str = None
52
+ chat_template_sha: str = None
53
+ start_time: float = None
54
+ end_time: float = None
55
+ total_evaluation_time_seconds: str = None
56
+
57
+ def __init__(self) -> None:
58
+ """Starts the evaluation timer."""
59
+ self.start_time = time.perf_counter()
60
+
61
+ @staticmethod
62
+ def _get_model_name(model_args: str) -> str:
63
+ """Extracts the model name from the model arguments."""
64
+
65
+ def extract_model_name(model_args: str, key: str) -> str:
66
+ """Extracts the model name from the model arguments using a key."""
67
+ args_after_key = model_args.split(key)[1]
68
+ return args_after_key.split(",")[0]
69
+
70
+ # order does matter, e.g. peft and delta are provided together with pretrained
71
+ prefixes = ["peft=", "delta=", "pretrained=", "model=", "path=", "engine="]
72
+ for prefix in prefixes:
73
+ if prefix in model_args:
74
+ return extract_model_name(model_args, prefix)
75
+ return ""
76
+
77
+ def log_experiment_args(
78
+ self,
79
+ model_source: str,
80
+ model_args: str,
81
+ system_instruction: str,
82
+ chat_template: str,
83
+ fewshot_as_multiturn: bool,
84
+ ) -> None:
85
+ """Logs model parameters and job ID."""
86
+ self.model_source = model_source
87
+ self.model_name = GeneralConfigTracker._get_model_name(model_args)
88
+ self.model_name_sanitized = sanitize_model_name(self.model_name)
89
+ self.system_instruction = system_instruction
90
+ self.system_instruction_sha = hash_string(system_instruction) if system_instruction else None
91
+ self.chat_template = chat_template
92
+ self.chat_template_sha = hash_string(chat_template) if chat_template else None
93
+ self.fewshot_as_multiturn = fewshot_as_multiturn
94
+
95
+ def log_end_time(self) -> None:
96
+ """Logs the end time of the evaluation and calculates the total evaluation time."""
97
+ self.end_time = time.perf_counter()
98
+ self.total_evaluation_time_seconds = str(self.end_time - self.start_time)
99
+
100
+
101
+ class EvaluationTracker:
102
+ """
103
+ Keeps track and saves relevant information of the evaluation process.
104
+ Compiles the data from trackers and writes it to files, which can be published to the Hugging Face hub if requested.
105
+ """
106
+
107
+ def __init__(
108
+ self,
109
+ output_path: str = None,
110
+ hub_results_org: str = "",
111
+ hub_repo_name: str = "",
112
+ details_repo_name: str = "",
113
+ results_repo_name: str = "",
114
+ push_results_to_hub: bool = False,
115
+ push_samples_to_hub: bool = False,
116
+ public_repo: bool = False,
117
+ token: str = "",
118
+ leaderboard_url: str = "",
119
+ point_of_contact: str = "",
120
+ gated: bool = False,
121
+ ) -> None:
122
+ """
123
+ Creates all the necessary loggers for evaluation tracking.
124
+
125
+ Args:
126
+ output_path (str): Path to save the results. If not provided, the results won't be saved.
127
+ hub_results_org (str): The Hugging Face organization to push the results to. If not provided, the results will be pushed to the owner of the Hugging Face token.
128
+ hub_repo_name (str): The name of the Hugging Face repository to push the results to. If not provided, the results will be pushed to `lm-eval-results`.
129
+ details_repo_name (str): The name of the Hugging Face repository to push the details to. If not provided, the results will be pushed to `lm-eval-results`.
130
+ result_repo_name (str): The name of the Hugging Face repository to push the results to. If not provided, the results will not be pushed and will be found in the details_hub_repo.
131
+ push_results_to_hub (bool): Whether to push the results to the Hugging Face hub.
132
+ push_samples_to_hub (bool): Whether to push the samples to the Hugging Face hub.
133
+ public_repo (bool): Whether to push the results to a public or private repository.
134
+ token (str): Token to use when pushing to the Hugging Face hub. This token should have write access to `hub_results_org`.
135
+ leaderboard_url (str): URL to the leaderboard on the Hugging Face hub on the dataset card.
136
+ point_of_contact (str): Contact information on the Hugging Face hub dataset card.
137
+ gated (bool): Whether to gate the repository.
138
+ """
139
+ self.general_config_tracker = GeneralConfigTracker()
140
+
141
+ self.output_path = output_path
142
+ self.push_results_to_hub = push_results_to_hub
143
+ self.push_samples_to_hub = push_samples_to_hub
144
+ self.public_repo = public_repo
145
+ self.leaderboard_url = leaderboard_url
146
+ self.point_of_contact = point_of_contact
147
+ self.api = HfApi(token=token) if token else None
148
+ self.gated_repo = gated
149
+
150
+ if not self.api and (push_results_to_hub or push_samples_to_hub):
151
+ raise ValueError("Hugging Face token is not defined, but 'push_results_to_hub' or 'push_samples_to_hub' is set to True. " "Please provide a valid Hugging Face token by setting the HF_TOKEN environment variable.")
152
+
153
+ if self.api and hub_results_org == "" and (push_results_to_hub or push_samples_to_hub):
154
+ hub_results_org = self.api.whoami()["name"]
155
+ eval_logger.warning(f"hub_results_org was not specified. Results will be pushed to '{hub_results_org}'.")
156
+
157
+ if hub_repo_name == "":
158
+ details_repo_name = details_repo_name if details_repo_name != "" else "lmms-eval-results"
159
+ results_repo_name = results_repo_name if results_repo_name != "" else details_repo_name
160
+ else:
161
+ details_repo_name = hub_repo_name
162
+ results_repo_name = hub_repo_name
163
+ eval_logger.warning("hub_repo_name was specified. Both details and results will be pushed to the same repository. Using hub_repo_name is no longer recommended, details_repo_name and results_repo_name should be used instead.")
164
+
165
+ self.details_repo = f"{hub_results_org}/{details_repo_name}"
166
+ self.details_repo_private = f"{hub_results_org}/{details_repo_name}-private"
167
+ self.results_repo = f"{hub_results_org}/{results_repo_name}"
168
+ self.results_repo_private = f"{hub_results_org}/{results_repo_name}-private"
169
+
170
+ def save_results_aggregated(
171
+ self,
172
+ results: dict,
173
+ samples: dict,
174
+ datetime_str: str,
175
+ ) -> None:
176
+ """
177
+ Saves the aggregated results and samples to the output path and pushes them to the Hugging Face hub if requested.
178
+
179
+ Args:
180
+ results (dict): The aggregated results to save.
181
+ samples (dict): The samples results to save.
182
+ datetime_str (str): The datetime string to use for the results file.
183
+ """
184
+ self.general_config_tracker.log_end_time()
185
+
186
+ if self.output_path:
187
+ try:
188
+ eval_logger.info("Saving results aggregated")
189
+
190
+ # calculate cumulative hash for each task - only if samples are provided
191
+ task_hashes = {}
192
+ if samples:
193
+ for task_name, task_samples in samples.items():
194
+ sample_hashes = [s["doc_hash"] + s["prompt_hash"] + s["target_hash"] for s in task_samples]
195
+ task_hashes[task_name] = hash_string("".join(sample_hashes))
196
+
197
+ # update initial results dict
198
+ results.update({"task_hashes": task_hashes})
199
+ results.update(asdict(self.general_config_tracker))
200
+ dumped = json.dumps(
201
+ results,
202
+ indent=2,
203
+ default=handle_non_serializable,
204
+ ensure_ascii=False,
205
+ )
206
+
207
+ path = Path(self.output_path if self.output_path else Path.cwd())
208
+ path = path.joinpath(self.general_config_tracker.model_name_sanitized)
209
+ path.mkdir(parents=True, exist_ok=True)
210
+
211
+ self.date_id = datetime_str.replace(":", "-")
212
+ file_results_aggregated = path.joinpath(f"{self.date_id}_results.json")
213
+ file_results_aggregated.open("w", encoding="utf-8").write(dumped)
214
+
215
+ if self.api and self.push_results_to_hub:
216
+ repo_id = self.results_repo if self.public_repo else self.results_repo_private
217
+ self.api.create_repo(
218
+ repo_id=repo_id,
219
+ repo_type="dataset",
220
+ private=not self.public_repo,
221
+ exist_ok=True,
222
+ )
223
+ self.api.upload_file(
224
+ repo_id=repo_id,
225
+ path_or_fileobj=str(path.joinpath(f"{self.date_id}_results.json")),
226
+ path_in_repo=os.path.join(
227
+ self.general_config_tracker.model_name,
228
+ f"{self.date_id}_results.json",
229
+ ),
230
+ repo_type="dataset",
231
+ commit_message=f"Adding aggregated results for {self.general_config_tracker.model_name}",
232
+ )
233
+ eval_logger.info("Successfully pushed aggregated results to the Hugging Face Hub. " f"You can find them at: {repo_id}")
234
+
235
+ except Exception as e:
236
+ eval_logger.warning("Could not save results aggregated")
237
+ eval_logger.info(repr(e))
238
+ else:
239
+ eval_logger.info("Output path not provided, skipping saving results aggregated")
240
+
241
+ def save_results_samples(
242
+ self,
243
+ task_name: str,
244
+ samples: dict,
245
+ ) -> None:
246
+ """
247
+ Saves the samples results to the output path and pushes them to the Hugging Face hub if requested.
248
+
249
+ Args:
250
+ task_name (str): The task name to save the samples for.
251
+ samples (dict): The samples results to save.
252
+ """
253
+ if self.output_path:
254
+ try:
255
+ eval_logger.info(f"Saving per-sample results for: {task_name}")
256
+
257
+ path = Path(self.output_path if self.output_path else Path.cwd())
258
+ path = path.joinpath(self.general_config_tracker.model_name_sanitized)
259
+ path.mkdir(parents=True, exist_ok=True)
260
+
261
+ file_results_samples = path.joinpath(f"{self.date_id}_samples_{task_name}.jsonl")
262
+
263
+ for sample in samples:
264
+ # we first need to sanitize arguments and resps
265
+ # otherwise we won't be able to load the dataset
266
+ # using the datasets library
267
+ arguments = {}
268
+ for key, value in enumerate(sample["arguments"][1]): # update metadata into args
269
+ arguments[key] = value
270
+
271
+ sample["input"] = sample["arguments"][0]
272
+ sample["resps"] = sanitize_list(sample["resps"])
273
+ sample["filtered_resps"] = sanitize_list(sample["filtered_resps"])
274
+ sample["arguments"] = arguments
275
+ sample["target"] = str(sample["target"])
276
+
277
+ sample_dump = (
278
+ json.dumps(
279
+ sample,
280
+ default=handle_non_serializable,
281
+ ensure_ascii=False,
282
+ )
283
+ + "\n"
284
+ )
285
+
286
+ with open(file_results_samples, "a", encoding="utf-8") as f:
287
+ f.write(sample_dump)
288
+
289
+ if self.api and self.push_samples_to_hub:
290
+ repo_id = self.details_repo if self.public_repo else self.details_repo_private
291
+ self.api.create_repo(
292
+ repo_id=repo_id,
293
+ repo_type="dataset",
294
+ private=not self.public_repo,
295
+ exist_ok=True,
296
+ )
297
+ try:
298
+ if self.gated_repo:
299
+ headers = build_hf_headers()
300
+ r = get_session().put(
301
+ url=f"https://huggingface.co/api/datasets/{repo_id}/settings",
302
+ headers=headers,
303
+ json={"gated": "auto"},
304
+ )
305
+ hf_raise_for_status(r)
306
+ except Exception as e:
307
+ eval_logger.warning("Could not gate the repository")
308
+ eval_logger.info(repr(e))
309
+ self.api.upload_folder(
310
+ repo_id=repo_id,
311
+ folder_path=str(path),
312
+ path_in_repo=self.general_config_tracker.model_name_sanitized,
313
+ repo_type="dataset",
314
+ commit_message=f"Adding samples results for {task_name} to {self.general_config_tracker.model_name}",
315
+ )
316
+ eval_logger.info(f"Successfully pushed sample results for task: {task_name} to the Hugging Face Hub. " f"You can find them at: {repo_id}")
317
+
318
+ except Exception as e:
319
+ eval_logger.warning("Could not save sample results")
320
+ eval_logger.info(repr(e))
321
+ else:
322
+ eval_logger.info("Output path not provided, skipping saving sample results")
323
+
324
+ def recreate_metadata_card(self) -> None:
325
+ """
326
+ Creates a metadata card for the evaluation results dataset and pushes it to the Hugging Face hub.
327
+ """
328
+
329
+ eval_logger.info("Recreating metadata card")
330
+ repo_id = self.details_repo if self.public_repo else self.details_repo_private
331
+
332
+ files_in_repo = self.api.list_repo_files(repo_id=repo_id, repo_type="dataset")
333
+ results_files = get_results_filenames(files_in_repo)
334
+ sample_files = get_sample_results_filenames(files_in_repo)
335
+
336
+ # Build a dictionary to store the latest evaluation datetime for:
337
+ # - Each tested model and its aggregated results
338
+ # - Each task and sample results, if existing
339
+ # i.e. {
340
+ # "org__model_name__gsm8k": "2021-09-01T12:00:00",
341
+ # "org__model_name__ifeval": "2021-09-01T12:00:00",
342
+ # "org__model_name__results": "2021-09-01T12:00:00"
343
+ # }
344
+ latest_task_results_datetime = defaultdict(lambda: datetime.min.isoformat())
345
+
346
+ for file_path in sample_files:
347
+ file_path = Path(file_path)
348
+ filename = file_path.name
349
+ model_name = file_path.parent
350
+ task_name = get_file_task_name(filename)
351
+ results_datetime = get_file_datetime(filename)
352
+ task_name_sanitized = sanitize_task_name(task_name)
353
+ # Results and sample results for the same model and task will have the same datetime
354
+ samples_key = f"{model_name}__{task_name_sanitized}"
355
+ results_key = f"{model_name}__results"
356
+ latest_datetime = max(
357
+ latest_task_results_datetime[samples_key],
358
+ results_datetime,
359
+ )
360
+ latest_task_results_datetime[samples_key] = latest_datetime
361
+ latest_task_results_datetime[results_key] = max(
362
+ latest_task_results_datetime[results_key],
363
+ latest_datetime,
364
+ )
365
+
366
+ # Create metadata card
367
+ card_metadata = MetadataConfigs()
368
+
369
+ # Add the latest aggregated results to the metadata card for easy access
370
+ for file_path in results_files:
371
+ file_path = Path(file_path)
372
+ results_filename = file_path.name
373
+ model_name = file_path.parent
374
+ eval_date = get_file_datetime(results_filename)
375
+ eval_date_sanitized = re.sub(r"[^\w\.]", "_", eval_date)
376
+ results_filename = Path("**") / Path(results_filename).name
377
+ config_name = f"{model_name}__results"
378
+ sanitized_last_eval_date_results = re.sub(r"[^\w\.]", "_", latest_task_results_datetime[config_name])
379
+
380
+ if eval_date_sanitized == sanitized_last_eval_date_results:
381
+ # Ensure that all results files are listed in the metadata card
382
+ current_results = card_metadata.get(config_name, {"data_files": []})
383
+ current_results["data_files"].append({"split": eval_date_sanitized, "path": [str(results_filename)]})
384
+ card_metadata[config_name] = current_results
385
+ # If the results file is the newest, update the "latest" field in the metadata card
386
+ card_metadata[config_name]["data_files"].append({"split": "latest", "path": [str(results_filename)]})
387
+
388
+ # Add the tasks details configs
389
+ for file_path in sample_files:
390
+ file_path = Path(file_path)
391
+ filename = file_path.name
392
+ model_name = file_path.parent
393
+ task_name = get_file_task_name(filename)
394
+ eval_date = get_file_datetime(filename)
395
+ task_name_sanitized = sanitize_task_name(task_name)
396
+ eval_date_sanitized = re.sub(r"[^\w\.]", "_", eval_date)
397
+ results_filename = Path("**") / Path(filename).name
398
+ config_name = f"{model_name}__{task_name_sanitized}"
399
+ sanitized_last_eval_date_results = re.sub(r"[^\w\.]", "_", latest_task_results_datetime[config_name])
400
+ if eval_date_sanitized == sanitized_last_eval_date_results:
401
+ # Ensure that all sample results files are listed in the metadata card
402
+ current_details_for_task = card_metadata.get(config_name, {"data_files": []})
403
+ current_details_for_task["data_files"].append({"split": eval_date_sanitized, "path": [str(results_filename)]})
404
+ card_metadata[config_name] = current_details_for_task
405
+ # If the samples results file is the newest, update the "latest" field in the metadata card
406
+ card_metadata[config_name]["data_files"].append({"split": "latest", "path": [str(results_filename)]})
407
+
408
+ # Get latest results and extract info to update metadata card examples
409
+ latest_datetime = max(latest_task_results_datetime.values())
410
+ latest_model_name = max(latest_task_results_datetime, key=lambda k: latest_task_results_datetime[k])
411
+ last_results_file = [f for f in results_files if latest_datetime.replace(":", "-") in f][0]
412
+ last_results_file_path = hf_hub_url(repo_id=repo_id, filename=last_results_file, repo_type="dataset")
413
+ latest_results_file = load_dataset("json", data_files=last_results_file_path, split="train")
414
+ results_dict = latest_results_file["results"][0]
415
+ new_dictionary = {"all": results_dict}
416
+ new_dictionary.update(results_dict)
417
+ results_string = json.dumps(new_dictionary, indent=4)
418
+
419
+ dataset_summary = "Dataset automatically created during the evaluation run of model "
420
+ if self.general_config_tracker.model_source == "hf":
421
+ dataset_summary += f"[{self.general_config_tracker.model_name}](https://huggingface.co/{self.general_config_tracker.model_name})\n"
422
+ else:
423
+ dataset_summary += f"{self.general_config_tracker.model_name}\n"
424
+ dataset_summary += (
425
+ f"The dataset is composed of {len(card_metadata)-1} configuration(s), each one corresponding to one of the evaluated task.\n\n"
426
+ f"The dataset has been created from {len(results_files)} run(s). Each run can be found as a specific split in each "
427
+ 'configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.\n\n'
428
+ 'An additional configuration "results" store all the aggregated results of the run.\n\n'
429
+ "To load the details from a run, you can for instance do the following:\n"
430
+ )
431
+ if self.general_config_tracker.model_source == "hf":
432
+ dataset_summary += "```python\nfrom datasets import load_dataset\n" f'data = load_dataset(\n\t"{repo_id}",\n\tname="{latest_model_name}",\n\tsplit="latest"\n)\n```\n\n'
433
+ dataset_summary += (
434
+ "## Latest results\n\n"
435
+ f'These are the [latest results from run {latest_datetime}]({last_results_file_path.replace("/resolve/", "/blob/")}) '
436
+ "(note that there might be results for other tasks in the repos if successive evals didn't cover the same tasks. "
437
+ 'You find each in the results and the "latest" split for each eval):\n\n'
438
+ f"```python\n{results_string}\n```"
439
+ )
440
+ card_data = DatasetCardData(
441
+ dataset_summary=dataset_summary,
442
+ repo_url=f"https://huggingface.co/{self.general_config_tracker.model_name}",
443
+ pretty_name=f"Evaluation run of {self.general_config_tracker.model_name}",
444
+ leaderboard_url=self.leaderboard_url,
445
+ point_of_contact=self.point_of_contact,
446
+ )
447
+ card_metadata.to_dataset_card_data(card_data)
448
+ card = DatasetCard.from_template(
449
+ card_data,
450
+ pretty_name=card_data.pretty_name,
451
+ )
452
+ card.push_to_hub(repo_id, repo_type="dataset")
LaViDa/eval/lmms_eval/loggers/utils.py ADDED
@@ -0,0 +1,127 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import re
3
+ import subprocess
4
+ from pathlib import Path
5
+ from typing import Any, Dict, Optional, Tuple, Union
6
+
7
+ import numpy as np
8
+ from loguru import logger
9
+ from torch.utils.collect_env import get_pretty_env_info
10
+ from transformers import __version__ as trans_version
11
+
12
+
13
+ def remove_none_pattern(input_string: str) -> Tuple[str, bool]:
14
+ """Remove the ',none' substring from the input_string if it exists at the end.
15
+
16
+ Args:
17
+ input_string (str): The input string from which to remove the ',none' substring.
18
+
19
+ Returns:
20
+ Tuple[str, bool]: A tuple containing the modified input_string with the ',none' substring removed
21
+ and a boolean indicating whether the modification was made (True) or not (False).
22
+ """
23
+ # Define the pattern to match ',none' at the end of the string
24
+ pattern = re.compile(r",none$")
25
+
26
+ # Use sub() to replace ',none' with an empty string
27
+ result = re.sub(pattern, "", input_string)
28
+
29
+ # check if the input_string changed
30
+ removed = result != input_string
31
+
32
+ return result, removed
33
+
34
+
35
+ def _handle_non_serializable(o: Any) -> Union[int, str, list]:
36
+ """Handle non-serializable objects by converting them to serializable types.
37
+
38
+ Args:
39
+ o (Any): The object to be handled.
40
+
41
+ Returns:
42
+ Union[int, str, list]: The converted object. If the object is of type np.int64 or np.int32,
43
+ it will be converted to int. If the object is of type set, it will be converted
44
+ to a list. Otherwise, it will be converted to str.
45
+ """
46
+ if isinstance(o, np.int64) or isinstance(o, np.int32):
47
+ return int(o)
48
+ elif isinstance(o, set):
49
+ return list(o)
50
+ else:
51
+ return str(o)
52
+
53
+
54
+ def get_commit_from_path(repo_path: Union[Path, str]) -> Optional[str]:
55
+ try:
56
+ git_folder = Path(repo_path, ".git")
57
+ if git_folder.is_file():
58
+ git_folder = Path(
59
+ git_folder.parent,
60
+ git_folder.read_text(encoding="utf-8").split("\n")[0].split(" ")[-1],
61
+ )
62
+ if Path(git_folder, "HEAD").exists():
63
+ head_name = Path(git_folder, "HEAD").read_text(encoding="utf-8").split("\n")[0].split(" ")[-1]
64
+ head_ref = Path(git_folder, head_name)
65
+ git_hash = head_ref.read_text(encoding="utf-8").replace("\n", "")
66
+ else:
67
+ git_hash = None
68
+ except Exception as err:
69
+ logger.debug(f"Failed to retrieve a Git commit hash from path: {str(repo_path)}. Error: {err}")
70
+ return None
71
+ return git_hash
72
+
73
+
74
+ def get_git_commit_hash():
75
+ """
76
+ Gets the git commit hash of your current repo (if it exists).
77
+ Source: https://github.com/EleutherAI/gpt-neox/blob/b608043be541602170bfcfb8ec9bf85e8a0799e0/megatron/neox_arguments/neox_args.py#L42
78
+ """
79
+ try:
80
+ git_hash = subprocess.check_output(["git", "describe", "--always"]).strip()
81
+ git_hash = git_hash.decode()
82
+ except (subprocess.CalledProcessError, FileNotFoundError):
83
+ # FileNotFoundError occurs when git not installed on system
84
+ git_hash = get_commit_from_path(os.getcwd()) # git hash of repo if exists
85
+ return git_hash
86
+
87
+
88
+ def add_env_info(storage: Dict[str, Any]):
89
+ try:
90
+ pretty_env_info = get_pretty_env_info()
91
+ except Exception as err:
92
+ pretty_env_info = str(err)
93
+ transformers_version = trans_version
94
+ upper_dir_commit = get_commit_from_path(Path(os.getcwd(), "..")) # git hash of upper repo if exists
95
+ added_info = {
96
+ "pretty_env_info": pretty_env_info,
97
+ "transformers_version": transformers_version,
98
+ "upper_git_hash": upper_dir_commit, # in case this repo is submodule
99
+ }
100
+ storage.update(added_info)
101
+
102
+
103
+ def add_tokenizer_info(storage: Dict[str, Any], lm):
104
+ if getattr(lm, "tokenizer", False):
105
+ try:
106
+ tokenizer_info = {
107
+ "tokenizer_pad_token": [
108
+ lm.tokenizer.pad_token,
109
+ str(lm.tokenizer.pad_token_id),
110
+ ],
111
+ "tokenizer_eos_token": [
112
+ lm.tokenizer.eos_token,
113
+ str(lm.tokenizer.eos_token_id),
114
+ ],
115
+ "tokenizer_bos_token": [
116
+ lm.tokenizer.bos_token,
117
+ str(lm.tokenizer.bos_token_id),
118
+ ],
119
+ "eot_token_id": getattr(lm, "eot_token_id", None),
120
+ "max_length": getattr(lm, "max_length", None),
121
+ }
122
+ storage.update(tokenizer_info)
123
+ except Exception as err:
124
+ logger.debug(f"Logging detailed tokenizer info failed with {err}, skipping...")
125
+ # seems gguf and textsynth do not have tokenizer
126
+ else:
127
+ logger.debug("LM does not have a 'tokenizer' attribute, not logging tokenizer metadata to results.")
LaViDa/eval/lmms_eval/loggers/wandb_logger.py ADDED
@@ -0,0 +1,315 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import copy
2
+ import json
3
+ import logging
4
+ from typing import Any, Dict, List, Literal, Tuple
5
+
6
+ import numpy as np
7
+ import pandas as pd
8
+ from loguru import logger
9
+ from packaging.version import Version
10
+
11
+ from lmms_eval.loggers.utils import _handle_non_serializable, remove_none_pattern
12
+
13
+
14
+ def get_wandb_printer() -> Literal["Printer"]:
15
+ """Returns a wandb printer instance for pretty stdout."""
16
+ from wandb.sdk.lib.printer import get_printer
17
+ from wandb.sdk.wandb_settings import Settings
18
+
19
+ printer = get_printer(Settings()._jupyter)
20
+ return printer
21
+
22
+
23
+ class WandbLogger:
24
+ def __init__(self, **kwargs) -> None:
25
+ """Attaches to wandb logger if already initialized. Otherwise, passes kwargs to wandb.init()
26
+
27
+ Args:
28
+ kwargs Optional[Any]: Arguments for configuration.
29
+
30
+ Parse and log the results returned from evaluator.simple_evaluate() with:
31
+ wandb_logger.post_init(results)
32
+ wandb_logger.log_eval_result()
33
+ wandb_logger.log_eval_samples(results["samples"])
34
+ """
35
+ try:
36
+ import wandb
37
+
38
+ assert Version(wandb.__version__) >= Version("0.13.6")
39
+ if Version(wandb.__version__) < Version("0.13.6"):
40
+ wandb.require("report-editing:v0")
41
+ except Exception as e:
42
+ logger.warning("To use the wandb reporting functionality please install wandb>=0.13.6.\n" "To install the latest version of wandb run `pip install wandb --upgrade`\n" f"{e}")
43
+
44
+ self.wandb_args: Dict[str, Any] = kwargs
45
+
46
+ # initialize a W&B run
47
+ if wandb.run is None:
48
+ self.run = wandb.init(**self.wandb_args)
49
+ else:
50
+ self.run = wandb.run
51
+
52
+ self.printer = get_wandb_printer()
53
+
54
+ def post_init(self, results: Dict[str, Any]) -> None:
55
+ self.results: Dict[str, Any] = copy.deepcopy(results)
56
+ self.task_names: List[str] = list(results.get("results", {}).keys())
57
+ self.group_names: List[str] = list(results.get("groups", {}).keys())
58
+
59
+ def _get_config(self) -> Dict[str, Any]:
60
+ """Get configuration parameters."""
61
+ self.task_configs = self.results.get("configs", {})
62
+ cli_configs = self.results.get("config", {})
63
+ configs = {
64
+ "task_configs": self.task_configs,
65
+ "cli_configs": cli_configs,
66
+ }
67
+
68
+ return configs
69
+
70
+ def _sanitize_results_dict(self) -> Tuple[Dict[str, str], Dict[str, Any]]:
71
+ """Sanitize the results dictionary."""
72
+ _results = copy.deepcopy(self.results.get("results", dict()))
73
+
74
+ # Remove None from the metric string name
75
+ tmp_results = copy.deepcopy(_results)
76
+ for task_name in self.task_names:
77
+ task_result = tmp_results.get(task_name, dict())
78
+ for metric_name, metric_value in task_result.items():
79
+ _metric_name, removed = remove_none_pattern(metric_name)
80
+ if removed:
81
+ _results[task_name][_metric_name] = metric_value
82
+ _results[task_name].pop(metric_name)
83
+
84
+ # remove string valued keys from the results dict
85
+ wandb_summary = {}
86
+ for task in self.task_names:
87
+ task_result = _results.get(task, dict())
88
+ for metric_name, metric_value in task_result.items():
89
+ if isinstance(metric_value, str):
90
+ wandb_summary[f"{task}/{metric_name}"] = metric_value
91
+
92
+ for summary_metric, summary_value in wandb_summary.items():
93
+ _task, _summary_metric = summary_metric.split("/")
94
+ _results[_task].pop(_summary_metric)
95
+
96
+ tmp_results = copy.deepcopy(_results)
97
+ for task_name, task_results in tmp_results.items():
98
+ for metric_name, metric_value in task_results.items():
99
+ _results[f"{task_name}/{metric_name}"] = metric_value
100
+ _results[task_name].pop(metric_name)
101
+ for task in self.task_names:
102
+ _results.pop(task)
103
+
104
+ return wandb_summary, _results
105
+
106
+ def _log_results_as_table(self) -> None:
107
+ """Generate and log evaluation results as a table to W&B."""
108
+ columns = [
109
+ "Version",
110
+ "Filter",
111
+ "num_fewshot",
112
+ "Metric",
113
+ "Value",
114
+ "Stderr",
115
+ ]
116
+
117
+ def make_table(columns: List[str], key: str = "results"):
118
+ import wandb
119
+
120
+ table = wandb.Table(columns=columns)
121
+ results = copy.deepcopy(self.results)
122
+
123
+ for k, dic in results.get(key).items():
124
+ if k in self.group_names and not key == "groups":
125
+ continue
126
+ version = results.get("versions").get(k)
127
+ if version == "N/A":
128
+ version = None
129
+ n = results.get("n-shot").get(k)
130
+
131
+ for (mf), v in dic.items():
132
+ m, _, f = mf.partition(",")
133
+ if m.endswith("_stderr"):
134
+ continue
135
+ if m == "alias":
136
+ continue
137
+
138
+ if m + "_stderr" + "," + f in dic:
139
+ se = dic[m + "_stderr" + "," + f]
140
+ if se != "N/A":
141
+ se = "%.4f" % se
142
+ table.add_data(*[k, version, f, n, m, str(v), str(se)])
143
+ else:
144
+ table.add_data(*[k, version, f, n, m, str(v), ""])
145
+
146
+ return table
147
+
148
+ # log the complete eval result to W&B Table
149
+ table = make_table(["Tasks"] + columns, "results")
150
+ self.run.log({"evaluation/eval_results": table})
151
+
152
+ if "groups" in self.results.keys():
153
+ table = make_table(["Groups"] + columns, "groups")
154
+ self.run.log({"evaluation/group_eval_results": table})
155
+
156
+ def _log_results_as_artifact(self) -> None:
157
+ """Log results as JSON artifact to W&B."""
158
+ import wandb
159
+
160
+ dumped = json.dumps(self.results, indent=2, default=_handle_non_serializable, ensure_ascii=False)
161
+ artifact = wandb.Artifact("results", type="eval_results")
162
+ with artifact.new_file("results.json", mode="w", encoding="utf-8") as f:
163
+ f.write(dumped)
164
+ self.run.log_artifact(artifact)
165
+
166
+ def log_eval_result(self) -> None:
167
+ """Log evaluation results to W&B."""
168
+ # Log configs to wandb
169
+ configs = self._get_config()
170
+ self.run.config.update(configs)
171
+
172
+ wandb_summary, self.wandb_results = self._sanitize_results_dict()
173
+ # update wandb.run.summary with items that were removed
174
+ self.run.summary.update(wandb_summary)
175
+ # Log the evaluation metrics to wandb
176
+ self.run.log(self.wandb_results)
177
+ # Log the evaluation metrics as W&B Table
178
+ self._log_results_as_table()
179
+ # Log the results dict as json to W&B Artifacts
180
+ self._log_results_as_artifact()
181
+
182
+ def _generate_dataset(self, data: List[Dict[str, Any]], config: Dict[str, Any]) -> pd.DataFrame:
183
+ """Generate a dataset from evaluation data.
184
+
185
+ Args:
186
+ data (List[Dict[str, Any]]): The data to generate a dataset for.
187
+ config (Dict[str, Any]): The configuration of the task.
188
+
189
+ Returns:
190
+ pd.DataFrame: A dataframe that is ready to be uploaded to W&B.
191
+ """
192
+ ids = [x["doc_id"] for x in data]
193
+ labels = [x["target"] for x in data]
194
+ instance = [""] * len(ids)
195
+ resps = [""] * len(ids)
196
+ filtered_resps = [""] * len(ids)
197
+ model_outputs = {}
198
+
199
+ metrics_list = config["metric_list"]
200
+ metrics = {}
201
+ for metric in metrics_list:
202
+ metric = metric.get("metric")
203
+ if metric in ["word_perplexity", "byte_perplexity", "bits_per_byte"]:
204
+ metrics[f"{metric}_loglikelihood"] = [x[metric][0] for x in data]
205
+ if metric in ["byte_perplexity", "bits_per_byte"]:
206
+ metrics[f"{metric}_bytes"] = [x[metric][1] for x in data]
207
+ else:
208
+ metrics[f"{metric}_words"] = [x[metric][1] for x in data]
209
+ else:
210
+ metrics[metric] = [x[metric] for x in data]
211
+
212
+ if config["output_type"] == "loglikelihood":
213
+ instance = [x["arguments"][0][0] for x in data]
214
+ labels = [x["arguments"][0][1] for x in data]
215
+ resps = [f'log probability of continuation is {x["resps"][0][0][0]} ' + "\n\n" + "continuation will {} generated with greedy sampling".format("not be" if not x["resps"][0][0][1] else "be") for x in data]
216
+ filtered_resps = [f'log probability of continuation is {x["filtered_resps"][0][0]} ' + "\n\n" + "continuation will {} generated with greedy sampling".format("not be" if not x["filtered_resps"][0][1] else "be") for x in data]
217
+ elif config["output_type"] == "multiple_choice":
218
+ instance = [x["arguments"][0][0] for x in data]
219
+ choices = ["\n".join([f"{idx}. {y[1]}" for idx, y in enumerate(x["arguments"])]) for x in data]
220
+ resps = [np.argmax([n[0][0] for n in x["resps"]]) for x in data]
221
+ filtered_resps = [np.argmax([n[0] for n in x["filtered_resps"]]) for x in data]
222
+ elif config["output_type"] == "loglikelihood_rolling":
223
+ instance = [x["arguments"][0][0] for x in data]
224
+ resps = [x["resps"][0][0] for x in data]
225
+ filtered_resps = [x["filtered_resps"][0] for x in data]
226
+ elif config["output_type"] == "generate_until":
227
+ instance = [x["arguments"][0][0] for x in data]
228
+ resps = [x["resps"][0][0] for x in data]
229
+ filtered_resps = [x["filtered_resps"][0] for x in data]
230
+
231
+ model_outputs["raw_predictions"] = resps
232
+ model_outputs["filtered_predictions"] = filtered_resps
233
+
234
+ df_data = {
235
+ "id": ids,
236
+ "data": instance,
237
+ }
238
+ if config["output_type"] == "multiple_choice":
239
+ df_data["choices"] = choices
240
+
241
+ tmp_data = {
242
+ "input_len": [len(x) for x in instance],
243
+ "labels": labels,
244
+ "output_type": config["output_type"],
245
+ }
246
+ df_data.update(tmp_data)
247
+ df_data.update(model_outputs)
248
+ df_data.update(metrics)
249
+
250
+ return pd.DataFrame(df_data)
251
+
252
+ def _log_samples_as_artifact(self, data: List[Dict[str, Any]], task_name: str) -> None:
253
+ import wandb
254
+
255
+ # log the samples as an artifact
256
+ dumped = json.dumps(
257
+ data,
258
+ indent=2,
259
+ default=_handle_non_serializable,
260
+ ensure_ascii=False,
261
+ )
262
+ artifact = wandb.Artifact(f"{task_name}", type="samples_by_task")
263
+ with artifact.new_file(f"{task_name}_eval_samples.json", mode="w", encoding="utf-8") as f:
264
+ f.write(dumped)
265
+ self.run.log_artifact(artifact)
266
+ # artifact.wait()
267
+
268
+ def log_eval_samples(self, samples: Dict[str, List[Dict[str, Any]]]) -> None:
269
+ """Log evaluation samples to W&B.
270
+
271
+ Args:
272
+ samples (Dict[str, List[Dict[str, Any]]]): Evaluation samples for each task.
273
+ """
274
+ task_names: List[str] = [x for x in self.task_names if x not in self.group_names]
275
+
276
+ ungrouped_tasks = []
277
+ tasks_by_groups = {}
278
+
279
+ for task_name in task_names:
280
+ group_names = self.task_configs[task_name].get("group", None)
281
+ if group_names:
282
+ if isinstance(group_names, str):
283
+ group_names = [group_names]
284
+
285
+ for group_name in group_names:
286
+ if not tasks_by_groups.get(group_name):
287
+ tasks_by_groups[group_name] = [task_name]
288
+ else:
289
+ tasks_by_groups[group_name].append(task_name)
290
+ else:
291
+ ungrouped_tasks.append(task_name)
292
+
293
+ for task_name in ungrouped_tasks:
294
+ eval_preds = samples[task_name]
295
+
296
+ # log the samples as a W&B Table
297
+ df = self._generate_dataset(eval_preds, self.task_configs.get(task_name))
298
+ self.run.log({f"{task_name}_eval_results": df})
299
+
300
+ # log the samples as a json file as W&B Artifact
301
+ self._log_samples_as_artifact(eval_preds, task_name)
302
+
303
+ for group, grouped_tasks in tasks_by_groups.items():
304
+ grouped_df = pd.DataFrame()
305
+ for task_name in grouped_tasks:
306
+ eval_preds = samples[task_name]
307
+ df = self._generate_dataset(eval_preds, self.task_configs.get(task_name))
308
+ df["group"] = group
309
+ df["task"] = task_name
310
+ grouped_df = pd.concat([grouped_df, df], ignore_index=True)
311
+
312
+ # log the samples as a json file as W&B Artifact
313
+ self._log_samples_as_artifact(eval_preds, task_name)
314
+
315
+ self.run.log({f"{group}_eval_results": grouped_df})
LaViDa/eval/lmms_eval/logging_utils.py ADDED
@@ -0,0 +1,368 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Code mostly from: https://github.com/EleutherAI/lm-evaluation-harness/pull/1339, credit to: https://github.com/ayulockin
2
+ import copy
3
+ import glob
4
+ import json
5
+ import os
6
+ import re
7
+ from datetime import datetime
8
+ from typing import Any, Dict, List, Literal, Tuple, Union
9
+
10
+ import numpy as np
11
+ import pandas as pd
12
+ import tenacity
13
+ from loguru import logger
14
+ from packaging.version import Version
15
+
16
+ from lmms_eval import utils
17
+
18
+ try:
19
+ import wandb
20
+
21
+ assert Version(wandb.__version__) >= Version("0.13.6")
22
+ if Version(wandb.__version__) < Version("0.13.6"):
23
+ wandb.require("report-editing:v0")
24
+ except Exception as e:
25
+ logger.warning("To use the wandb reporting functionality please install wandb>=0.13.6.\n" "To install the latest version of wandb run `pip install wandb --upgrade`\n" f"{e}")
26
+
27
+
28
+ def remove_none_pattern(input_string):
29
+ # Define the pattern to match ',none' at the end of the string
30
+ pattern = re.compile(r",none$")
31
+
32
+ # Use sub() to replace ',none' with an empty string
33
+ result = re.sub(pattern, "", input_string)
34
+
35
+ # check if the input_string changed
36
+ removed = result != input_string
37
+
38
+ return result, removed
39
+
40
+
41
+ def _handle_non_serializable(o: Any) -> Union[int, str, list]:
42
+ """Handle non-serializable objects by converting them to serializable types.
43
+
44
+ Args:
45
+ o (Any): The object to be handled.
46
+
47
+ Returns:
48
+ Union[int, str, list]: The converted object. If the object is of type np.int64 or np.int32,
49
+ it will be converted to int. If the object is of type set, it will be converted
50
+ to a list. Otherwise, it will be converted to str.
51
+ """
52
+ if isinstance(o, np.int64) or isinstance(o, np.int32):
53
+ return int(o)
54
+ elif isinstance(o, set):
55
+ return list(o)
56
+ else:
57
+ return str(o)
58
+
59
+
60
+ def get_wandb_printer() -> Literal["Printer"]:
61
+ """Returns a wandb printer instance for pretty stdout."""
62
+ from wandb.sdk.lib.printer import get_printer
63
+ from wandb.sdk.wandb_settings import Settings
64
+
65
+ printer = get_printer(Settings()._jupyter)
66
+ return printer
67
+
68
+
69
+ # class WandbLogger:
70
+ class WandbLogger:
71
+ def __init__(self, args):
72
+ self.wandb_args = utils.simple_parse_args_string(args.wandb_args)
73
+ self.args = args
74
+ self.all_args_dict = vars(args)
75
+ self.printer = get_wandb_printer()
76
+ try:
77
+ self.init_run()
78
+ except Exception as e:
79
+ logger.warning(f"Failed to initialize W&B run: {e}")
80
+ os.environ["WANDB_MODE"] = "offline"
81
+ self.init_run()
82
+
83
+ def finish(self):
84
+ self.run.finish()
85
+
86
+ @tenacity.retry(wait=tenacity.wait_fixed(5), stop=tenacity.stop_after_attempt(5))
87
+ def init_run(self):
88
+ if "name" not in self.wandb_args:
89
+ if "config" in self.all_args_dict and self.all_args_dict["config"] != "":
90
+ self.wandb_args["name"] = self.all_args_dict["config"].split("/")[-1].replace(".yaml", "") + "/" + self.args.log_samples_suffix
91
+ else:
92
+ task_names = self.args.tasks.replace(",", "/")
93
+ self.wandb_args["name"] = f"{self.args.model}/<{task_names}>/{self.args.log_samples_suffix}"
94
+ if self.args.num_fewshot:
95
+ self.wandb_args["name"] += f"_{self.args.num_fewshot}shot"
96
+ if "project" not in self.wandb_args:
97
+ self.wandb_args["project"] = "lmms-eval"
98
+ # initialize a W&B run
99
+ self.run = wandb.init(**self.wandb_args)
100
+
101
+ def post_init(self, results: Dict[str, Any]) -> None:
102
+ self.results: Dict[str, Any] = copy.deepcopy(results)
103
+ self.task_names: List[str] = list(results.get("results", {}).keys())
104
+ self.group_names: List[str] = list(results.get("groups", {}).keys())
105
+
106
+ def _get_config(self) -> Dict[str, Any]:
107
+ """Get configuration parameters."""
108
+ self.task_configs = self.results.get("configs", {})
109
+ cli_configs = self.results.get("config", {})
110
+ configs = {
111
+ "task_configs": self.task_configs,
112
+ "cli_configs": cli_configs,
113
+ }
114
+
115
+ return configs
116
+
117
+ def _sanitize_results_dict(self) -> Tuple[Dict[str, str], Dict[str, Any]]:
118
+ """Sanitize the results dictionary."""
119
+ _results = copy.deepcopy(self.results.get("results", dict()))
120
+ _results["model_configs"] = self.results.get("model_configs", dict())
121
+
122
+ # Remove None from the metric string name
123
+ tmp_results = copy.deepcopy(_results)
124
+ for task_name in self.task_names:
125
+ task_result = tmp_results.get(task_name, dict())
126
+ for metric_name, metric_value in task_result.items():
127
+ _metric_name, removed = remove_none_pattern(metric_name)
128
+ if removed:
129
+ _results[task_name][_metric_name] = metric_value
130
+ _results[task_name].pop(metric_name)
131
+
132
+ # remove string valued keys from the results dict
133
+ wandb_summary = {}
134
+ for task in self.task_names:
135
+ task_result = _results.get(task, dict())
136
+ for metric_name, metric_value in task_result.items():
137
+ if isinstance(metric_value, str):
138
+ wandb_summary[f"{task}/{metric_name}"] = metric_value
139
+
140
+ wandb_summary["model_configs"] = self.results.get("model_configs", dict())
141
+ for summary_metric, summary_value in wandb_summary.items():
142
+ if summary_metric != "model_configs":
143
+ _task, _summary_metric = summary_metric.split("/")
144
+ _results[_task].pop(_summary_metric)
145
+
146
+ tmp_results = copy.deepcopy(_results)
147
+ for task_name, task_results in tmp_results.items():
148
+ if task_name != "model_configs":
149
+ for metric_name, metric_value in task_results.items():
150
+ _results[f"{task_name}/{metric_name}"] = metric_value
151
+ _results[task_name].pop(metric_name)
152
+ for task in self.task_names:
153
+ _results.pop(task)
154
+
155
+ return wandb_summary, _results
156
+
157
+ def _log_results_as_table(self) -> None:
158
+ """Generate and log evaluation results as a table to W&B."""
159
+ columns = [
160
+ "Model",
161
+ "Args",
162
+ "Tasks",
163
+ "Version",
164
+ "Filter",
165
+ "num_fewshot",
166
+ "Metric",
167
+ "Value",
168
+ "Stderr",
169
+ ]
170
+
171
+ def make_table(columns: List[str], key: str = "results"):
172
+ table = wandb.Table(columns=columns)
173
+ results = copy.deepcopy(self.results)
174
+
175
+ model_name = results.get("model_configs").get("model")
176
+ model_args = results.get("model_configs").get("model_args")
177
+
178
+ for k, dic in results.get(key).items():
179
+ if k in self.group_names and not key == "groups":
180
+ continue
181
+ version = results.get("versions").get(k)
182
+ if version == "N/A":
183
+ version = None
184
+ n = results.get("n-shot").get(k)
185
+
186
+ for (mf), v in dic.items():
187
+ m, _, f = mf.partition(",")
188
+ if m.endswith("_stderr"):
189
+ continue
190
+ if m == "alias":
191
+ continue
192
+
193
+ if m + "_stderr" + "," + f in dic:
194
+ se = dic[m + "_stderr" + "," + f]
195
+ if se != "N/A":
196
+ se = "%.4f" % se
197
+ data = [model_name, model_args, k, version, f, n, m, str(v), str(se)]
198
+ if key == "groups":
199
+ data = [self.group_names] + data
200
+ table.add_data(*data)
201
+ else:
202
+ data = [model_name, model_args, k, version, f, n, m, str(v), ""]
203
+ if key == "groups":
204
+ data = [self.group_names] + data
205
+ table.add_data(*data)
206
+
207
+ return table
208
+
209
+ # log the complete eval result to W&B Table
210
+ table = make_table(columns, "results")
211
+ self.run.log({"evaluation/eval_results": table})
212
+
213
+ if "groups" in self.results.keys():
214
+ table = make_table(["Groups"] + columns, "groups")
215
+ self.run.log({"evaluation/group_eval_results": table})
216
+
217
+ def _log_results_as_artifact(self) -> None:
218
+ """Log results as JSON artifact to W&B."""
219
+ dumped = json.dumps(self.results, indent=2, default=_handle_non_serializable, ensure_ascii=False)
220
+ artifact = wandb.Artifact("results", type="eval_results")
221
+ with artifact.new_file("results.json", mode="w", encoding="utf-8") as f:
222
+ f.write(dumped)
223
+ self.run.log_artifact(artifact)
224
+
225
+ def log_eval_result(self) -> None:
226
+ """Log evaluation results to W&B."""
227
+ # Log configs to wandb
228
+ configs = self._get_config()
229
+ self.run.config.update(configs, allow_val_change=True)
230
+
231
+ wandb_summary, self.wandb_results = self._sanitize_results_dict()
232
+ # update wandb.run.summary with items that were removed
233
+ self.run.summary.update(wandb_summary)
234
+ # Log the evaluation metrics to wandb
235
+ self.run.log(self.wandb_results)
236
+ # Log the evaluation metrics as W&B Table
237
+ self._log_results_as_table()
238
+ # Log the results dict as json to W&B Artifacts
239
+ self._log_results_as_artifact()
240
+
241
+ def _generate_dataset(self, data: List[Dict[str, Any]], config: Dict[str, Any]) -> pd.DataFrame:
242
+ """Generate a dataset from evaluation data.
243
+
244
+ Args:
245
+ data (List[Dict[str, Any]]): The data to generate a dataset for.
246
+ config (Dict[str, Any]): The configuration of the task.
247
+
248
+ Returns:
249
+ pd.DataFrame: A dataframe that is ready to be uploaded to W&B.
250
+ """
251
+ ids = [x["doc_id"] for x in data]
252
+ labels = [x["target"] for x in data]
253
+ instance = [""] * len(ids)
254
+ resps = [""] * len(ids)
255
+ filtered_resps = [""] * len(ids)
256
+ model_outputs = {}
257
+
258
+ metrics_list = config["metric_list"]
259
+ metrics = {}
260
+ for metric in metrics_list:
261
+ metric = metric.get("metric")
262
+ if metric in ["word_perplexity", "byte_perplexity", "bits_per_byte"]:
263
+ metrics[f"{metric}_loglikelihood"] = [x[metric][0] for x in data]
264
+ if metric in ["byte_perplexity", "bits_per_byte"]:
265
+ metrics[f"{metric}_bytes"] = [x[metric][1] for x in data]
266
+ else:
267
+ metrics[f"{metric}_words"] = [x[metric][1] for x in data]
268
+ else:
269
+ metrics[metric] = [x[metric] for x in data]
270
+
271
+ if config["output_type"] == "loglikelihood":
272
+ instance = [x["arguments"][0][0] for x in data]
273
+ labels = [x["arguments"][0][1] for x in data]
274
+ resps = [f'log probability of continuation is {x["resps"][0][0][0]} ' + "\n\n" + "continuation will {} generated with greedy sampling".format("not be" if not x["resps"][0][0][1] else "be") for x in data]
275
+ filtered_resps = [f'log probability of continuation is {x["filtered_resps"][0][0]} ' + "\n\n" + "continuation will {} generated with greedy sampling".format("not be" if not x["filtered_resps"][0][1] else "be") for x in data]
276
+ elif config["output_type"] == "multiple_choice":
277
+ instance = [x["arguments"][0][0] for x in data]
278
+ choices = ["\n".join([f"{idx}. {y[1]}" for idx, y in enumerate(x["arguments"])]) for x in data]
279
+ resps = [np.argmax([n[0][0] for n in x["resps"]]) for x in data]
280
+ filtered_resps = [np.argmax([n[0] for n in x["filtered_resps"]]) for x in data]
281
+ elif "generate_until" in config["output_type"]:
282
+ instance = [x["arguments"][0][0] for x in data]
283
+ resps = [x["resps"][0][0] for x in data]
284
+ filtered_resps = [x["filtered_resps"][0] for x in data]
285
+
286
+ model_outputs["raw_predictions"] = resps
287
+ model_outputs["filtered_predictions"] = filtered_resps
288
+
289
+ df_data = {
290
+ "id": ids,
291
+ "data": instance,
292
+ }
293
+ if config["output_type"] == "multiple_choice":
294
+ df_data["choices"] = choices
295
+
296
+ tmp_data = {
297
+ "input_len": [len(x) for x in instance],
298
+ "labels": labels,
299
+ "output_type": config["output_type"],
300
+ }
301
+ df_data.update(tmp_data)
302
+ df_data.update(model_outputs)
303
+ df_data.update(metrics)
304
+
305
+ return pd.DataFrame(df_data)
306
+
307
+ def _log_samples_as_artifact(self, data: List[Dict[str, Any]], task_name: str) -> None:
308
+ # log the samples as an artifact
309
+ dumped = json.dumps(
310
+ data,
311
+ indent=2,
312
+ default=_handle_non_serializable,
313
+ ensure_ascii=False,
314
+ )
315
+ artifact = wandb.Artifact(f"{task_name}", type="samples_by_task")
316
+ with artifact.new_file(f"{task_name}_eval_samples.json", mode="w", encoding="utf-8") as f:
317
+ f.write(dumped)
318
+ self.run.log_artifact(artifact)
319
+ # artifact.wait()
320
+
321
+ def log_eval_samples(self, samples: Dict[str, List[Dict[str, Any]]]) -> None:
322
+ """Log evaluation samples to W&B.
323
+
324
+ Args:
325
+ samples (Dict[str, List[Dict[str, Any]]]): Evaluation samples for each task.
326
+ """
327
+ task_names: List[str] = [x for x in self.task_names if x not in self.group_names]
328
+
329
+ ungrouped_tasks = []
330
+ tasks_by_groups = {}
331
+
332
+ for task_name in task_names:
333
+ group_names = self.task_configs[task_name].get("group", None)
334
+ if group_names:
335
+ if isinstance(group_names, str):
336
+ group_names = [group_names]
337
+
338
+ for group_name in group_names:
339
+ if not tasks_by_groups.get(group_name):
340
+ tasks_by_groups[group_name] = [task_name]
341
+ else:
342
+ tasks_by_groups[group_name].append(task_name)
343
+ else:
344
+ ungrouped_tasks.append(task_name)
345
+
346
+ for task_name in ungrouped_tasks:
347
+ eval_preds = samples[task_name]
348
+
349
+ # log the samples as a W&B Table
350
+ df = self._generate_dataset(eval_preds, self.task_configs.get(task_name))
351
+ self.run.log({f"{task_name}_eval_results": df})
352
+
353
+ # log the samples as a json file as W&B Artifact
354
+ self._log_samples_as_artifact(eval_preds, task_name)
355
+
356
+ for group, grouped_tasks in tasks_by_groups.items():
357
+ grouped_df = pd.DataFrame()
358
+ for task_name in grouped_tasks:
359
+ eval_preds = samples[task_name]
360
+ df = self._generate_dataset(eval_preds, self.task_configs.get(task_name))
361
+ df["group"] = group
362
+ df["task"] = task_name
363
+ grouped_df = pd.concat([grouped_df, df], ignore_index=True)
364
+
365
+ # log the samples as a json file as W&B Artifact
366
+ self._log_samples_as_artifact(eval_preds, task_name)
367
+
368
+ self.run.log({f"{group}_eval_results": grouped_df})
LaViDa/eval/lmms_eval/models/egogpt.py ADDED
@@ -0,0 +1,472 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import copy
2
+ import json
3
+ import logging
4
+ import math
5
+ import re
6
+ import warnings
7
+ from datetime import timedelta
8
+ from typing import List, Optional, Tuple, Union
9
+
10
+ import numpy as np
11
+ import PIL
12
+ import torch
13
+ import transformers
14
+ from accelerate import Accelerator, DistributedType, InitProcessGroupKwargs
15
+ from accelerate.state import AcceleratorState
16
+ from decord import VideoReader, cpu
17
+ from packaging import version
18
+ from tqdm import tqdm
19
+ from transformers import AutoConfig
20
+
21
+ from lmms_eval import utils
22
+ from lmms_eval.api.instance import Instance
23
+ from lmms_eval.api.model import lmms
24
+ from lmms_eval.api.registry import register_model
25
+
26
+ # Suppress warnings
27
+ warnings.filterwarnings("ignore")
28
+
29
+ # Configure logging
30
+ eval_logger = logging.getLogger("lmms-eval")
31
+
32
+ # Enable TF32 for CUDA
33
+ torch.backends.cuda.matmul.allow_tf32 = True
34
+
35
+ # Import LLaVA modules
36
+ try:
37
+ import copy
38
+ import os
39
+ import re
40
+ import sys
41
+ import warnings
42
+
43
+ import numpy as np
44
+ import requests
45
+ import soundfile as sf
46
+ import torch
47
+ import whisper
48
+ from decord import VideoReader, cpu
49
+ from egogpt.constants import (
50
+ DEFAULT_IMAGE_TOKEN,
51
+ DEFAULT_SPEECH_TOKEN,
52
+ IGNORE_INDEX,
53
+ IMAGE_TOKEN_INDEX,
54
+ SPEECH_TOKEN_INDEX,
55
+ )
56
+ from egogpt.conversation import SeparatorStyle, conv_templates
57
+ from egogpt.mm_utils import get_model_name_from_path, process_images
58
+ from egogpt.model.builder import load_pretrained_model
59
+ from PIL import Image
60
+ from scipy.signal import resample
61
+ except ImportError as e:
62
+ eval_logger.debug(f"egogpt is not installed. Please install egogpt to use this model.\nError: {e}")
63
+
64
+
65
+ # Determine best attention implementation
66
+ if version.parse(torch.__version__) >= version.parse("2.1.2"):
67
+ best_fit_attn_implementation = "sdpa"
68
+ else:
69
+ best_fit_attn_implementation = "eager"
70
+
71
+
72
+ @register_model("egogpt")
73
+ class EgoGPT(lmms):
74
+ """
75
+ EgoGPT Model
76
+ """
77
+
78
+ def __init__(
79
+ self,
80
+ pretrained: str = "checkpoints/egogpt_IT_12k_1126_zero3",
81
+ truncation: Optional[bool] = True,
82
+ device: Optional[str] = "cuda:0",
83
+ batch_size: Optional[Union[int, str]] = 1,
84
+ model_name: Optional[str] = None,
85
+ attn_implementation: Optional[str] = best_fit_attn_implementation,
86
+ device_map: Optional[str] = "cuda:0",
87
+ conv_template: Optional[str] = "qwen_1_5",
88
+ use_cache: Optional[bool] = True,
89
+ truncate_context: Optional[bool] = False, # whether to truncate the context in generation, set it False for LLaVA-1.6
90
+ customized_config: Optional[str] = None, # ends in json
91
+ max_frames_num: Optional[int] = 32,
92
+ mm_spatial_pool_stride: Optional[int] = 2,
93
+ mm_spatial_pool_mode: Optional[str] = "bilinear",
94
+ token_strategy: Optional[str] = "single", # could be "single" or "multiple", "multiple" denotes adding multiple <image> tokens for each frame
95
+ video_decode_backend: str = "decord",
96
+ **kwargs,
97
+ ) -> None:
98
+ super().__init__()
99
+ # Do not use kwargs for now
100
+ assert kwargs == {}, f"Unexpected kwargs: {kwargs}"
101
+
102
+ accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52))
103
+ accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs])
104
+ if accelerator.num_processes > 1:
105
+ self._device = torch.device(f"cuda:{accelerator.local_process_index}")
106
+ self.device_map = f"cuda:{accelerator.local_process_index}"
107
+ elif accelerator.num_processes == 1 and device_map == "auto":
108
+ self._device = torch.device(device)
109
+ self.device_map = device_map
110
+ else:
111
+ self._device = torch.device(f"cuda:{accelerator.local_process_index}")
112
+ self.device_map = f"cuda:{accelerator.local_process_index}"
113
+
114
+ egogpt_model_args = {}
115
+ if attn_implementation is not None:
116
+ egogpt_model_args["attn_implementation"] = attn_implementation
117
+
118
+ self.pretrained = pretrained
119
+ self.token_strategy = token_strategy
120
+ self.max_frames_num = max_frames_num
121
+ self.mm_spatial_pool_stride = mm_spatial_pool_stride
122
+ self.mm_spatial_pool_mode = mm_spatial_pool_mode
123
+ self.video_decode_backend = video_decode_backend
124
+ # Try to load the model with the multimodal argument
125
+ self._tokenizer, self._model, self._max_length = load_pretrained_model(pretrained, device_map=self.device_map, **egogpt_model_args)
126
+ self._image_processor = self._model.get_vision_tower().image_processor
127
+ self._config = self._model.config
128
+ self.model.eval()
129
+ self.truncation = truncation
130
+ self.batch_size_per_gpu = int(batch_size)
131
+ self.conv_template = conv_template
132
+ self.use_cache = use_cache
133
+ self.truncate_context = truncate_context
134
+ assert self.batch_size_per_gpu == 1
135
+
136
+ if accelerator.num_processes > 1:
137
+ assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported."
138
+ # If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model
139
+ # Also, you have to select zero stage 0 (equivalent to DDP) in order to make the prepare model works
140
+ # I tried to set different parameters in the kwargs to let default zero 2 stage works, but it didn't work.
141
+ if accelerator.distributed_type == DistributedType.DEEPSPEED:
142
+ kwargs = {
143
+ "train_micro_batch_size_per_gpu": self.batch_size_per_gpu,
144
+ "train_batch_size": self.batch_size_per_gpu * accelerator.num_processes,
145
+ }
146
+ AcceleratorState().deepspeed_plugin.deepspeed_config_process(must_match=True, **kwargs)
147
+ eval_logger.info("Detected that you are using DistributedType.DEEPSPEED. Make sure you run `accelerate config` and set zero stage to 0")
148
+
149
+ if accelerator.distributed_type == DistributedType.FSDP or accelerator.distributed_type == DistributedType.DEEPSPEED:
150
+ self._model = accelerator.prepare(self.model)
151
+ else:
152
+ self._model = accelerator.prepare_model(self.model, evaluation_mode=True)
153
+ self.accelerator = accelerator
154
+ if self.accelerator.is_local_main_process:
155
+ eval_logger.info(f"Using {accelerator.num_processes} devices with data parallelism")
156
+ self._rank = self.accelerator.local_process_index
157
+ self._world_size = self.accelerator.num_processes
158
+
159
+ elif accelerator.num_processes == 1 and device_map == "auto":
160
+ eval_logger.info(f"Using {accelerator.num_processes} devices with tensor parallelism")
161
+ self._rank = 0
162
+ self._world_size = 1
163
+
164
+ else:
165
+ eval_logger.info(f"Using single device: {self._device}")
166
+ self.model.to(self._device)
167
+ self._rank = 0
168
+ self._world_size = 1
169
+
170
+ @property
171
+ def config(self):
172
+ # return the associated transformers.AutoConfig for the given pretrained model.
173
+ return self._config
174
+
175
+ @property
176
+ def tokenizer(self):
177
+ return self._tokenizer
178
+
179
+ @property
180
+ def model(self):
181
+ # returns the model, unwrapping it if using Accelerate
182
+ if hasattr(self, "accelerator"):
183
+ return self.accelerator.unwrap_model(self._model)
184
+ else:
185
+ return self._model
186
+
187
+ @property
188
+ def eot_token_id(self):
189
+ # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
190
+ return self.tokenizer.eos_token_id
191
+
192
+ @property
193
+ def max_length(self):
194
+ return self._max_length
195
+
196
+ def pad_sequence(self, input_ids, batch_first, padding_value):
197
+ if self.tokenizer.padding_side == "left":
198
+ input_ids = [torch.flip(_input_ids, [0]) for _input_ids in input_ids]
199
+ input_ids = torch.nn.utils.rnn.pad_sequence(input_ids, batch_first=batch_first, padding_value=padding_value)
200
+ if self.tokenizer.padding_side == "left":
201
+ input_ids = torch.flip(input_ids, [1])
202
+ return input_ids
203
+
204
+ @property
205
+ def batch_size(self):
206
+ return self.batch_size_per_gpu
207
+
208
+ @property
209
+ def device(self):
210
+ return self._device
211
+
212
+ @property
213
+ def rank(self):
214
+ return self._rank
215
+
216
+ @property
217
+ def world_size(self):
218
+ return self._world_size
219
+
220
+ def tok_encode(self, string: str, left_truncate_len=None, add_special_tokens=None) -> List[int]:
221
+ """ """
222
+ add_special_tokens = False if add_special_tokens is None else add_special_tokens
223
+ encoding = self.tokenizer.encode(string, add_special_tokens=add_special_tokens)
224
+ # left-truncate the encoded context to be at most `left_truncate_len` tokens long
225
+ if left_truncate_len:
226
+ encoding = encoding[-left_truncate_len:]
227
+ return encoding
228
+
229
+ def tok_decode(self, tokens):
230
+ try:
231
+ return self.tokenizer.decode(tokens)
232
+ except:
233
+ return self.tokenizer.decode([tokens])
234
+
235
+ def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
236
+ raise NotImplementedError("Loglikelihood is not implemented for EgoGPT")
237
+
238
+ def flatten(self, input):
239
+ new_list = []
240
+ for i in input:
241
+ for j in i:
242
+ new_list.append(j)
243
+ return new_list
244
+
245
+ def split_text(self, text, keywords):
246
+ pattern = "(" + "|".join(map(re.escape, keywords)) + ")"
247
+ parts = re.split(pattern, text)
248
+ parts = [part for part in parts if part]
249
+ return parts
250
+
251
+ def load_video(self, video_path=None, audio_path=None, max_frames_num=16, fps=1, task_name=None):
252
+ if audio_path is not None:
253
+ speech, sample_rate = sf.read(audio_path)
254
+ if sample_rate != 16000:
255
+ target_length = int(len(speech) * 16000 / sample_rate)
256
+ speech = resample(speech, target_length)
257
+ if speech.ndim > 1:
258
+ speech = np.mean(speech, axis=1)
259
+ # max_length = 480000
260
+ speech = whisper.pad_or_trim(speech.astype(np.float32))
261
+ speech = whisper.log_mel_spectrogram(speech, n_mels=128).permute(1, 0)
262
+ speech_lengths = torch.LongTensor([speech.shape[0]])
263
+ else:
264
+ speech = torch.zeros(3000, 128)
265
+ speech_lengths = torch.LongTensor([3000])
266
+
267
+ vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)
268
+ total_frame_num = len(vr)
269
+ avg_fps = round(vr.get_avg_fps() / fps)
270
+ frame_idx = [i for i in range(0, total_frame_num, avg_fps)]
271
+ frame_time = [i / avg_fps for i in frame_idx]
272
+
273
+ if max_frames_num > 0:
274
+ if len(frame_idx) > max_frames_num:
275
+ uniform_sampled_frames = np.linspace(0, total_frame_num - 1, max_frames_num, dtype=int)
276
+ frame_idx = uniform_sampled_frames.tolist()
277
+ if task_name == "egoplan":
278
+ # add current ovservation frame
279
+ frame_idx.append(total_frame_num - 1)
280
+ video = vr.get_batch(frame_idx).asnumpy()
281
+ return video, speech, speech_lengths
282
+
283
+ def generate_until(self, requests: List[Instance]) -> List[str]:
284
+ res = []
285
+
286
+ def _collate(x):
287
+ # the negative sign on len(toks) sorts descending - this has a few advantages:
288
+ # - time estimates will always be over not underestimates, which is more useful for planning
289
+ # - to know the size of a batch when going through the list, you know the first one is always the batch
290
+ # padded context length. this is useful to simplify the batching logic and more importantly to make
291
+ # automatic adaptive batches much much easier to implement
292
+ # - any OOMs will happen right away rather than near the end
293
+ toks = self.tok_encode(x[0])
294
+ return -len(toks), x[0]
295
+
296
+ # we group requests by their generation_kwargs,
297
+ # so that we don't try to execute e.g. greedy sampling and temp=0.8 sampling
298
+ # in the same batch.
299
+ metadata = requests[0].metadata
300
+ re_ords = utils.Collator([reg.args for reg in requests], _collate, grouping=True)
301
+ chunks = re_ords.get_batched(n=self.batch_size, batch_fn=None)
302
+ num_iters = len(requests) // self.batch_size if len(requests) % self.batch_size == 0 else len(requests) // self.batch_size + 1
303
+ pbar = tqdm(total=num_iters, disable=(self.rank != 0), desc="Model Responding")
304
+
305
+ origin_image_aspect_ratio = getattr(self._config, "image_aspect_ratio", None)
306
+
307
+ for chunk in chunks:
308
+ batched_contexts, all_gen_kwargs, batched_doc_to_visual, batched_doc_id, batched_task, batched_split = zip(*chunk)
309
+ task = batched_task[0]
310
+ split = batched_split[0]
311
+ batched_visuals = [batched_doc_to_visual[0](self.task_dict[task][split][ids]) for ids in batched_doc_id] # [B, N]
312
+ assert len(batched_visuals) == 1
313
+
314
+ # we assume all gen kwargs in the batch are the same
315
+ # this is safe to assume because the `grouper` object ensures it.
316
+ gen_kwargs = all_gen_kwargs[0]
317
+ if "until" in gen_kwargs:
318
+ gen_kwargs.pop("until")
319
+
320
+ question_input = []
321
+ # import ipdb; ipdb.set_trace()
322
+ for visual, context in zip(batched_visuals, batched_contexts):
323
+ if origin_image_aspect_ratio is not None and self._config.image_aspect_ratio != origin_image_aspect_ratio:
324
+ self._config.image_aspect_ratio = origin_image_aspect_ratio
325
+ eval_logger.info(f"Resetting image aspect ratio to {origin_image_aspect_ratio}")
326
+
327
+ if visual is None or visual == []: # for text-only tasks.
328
+ visual = None
329
+ task_type = "text"
330
+ placeholder_count = 0
331
+ image_tensor = None
332
+ else:
333
+ if len(visual) > 1 or "image_aspect_ratio" not in self._config.__dict__: # for multi image case, we treat per image aspect ratio as "pad" by default.
334
+ self._config.image_aspect_ratio = getattr(gen_kwargs, "image_aspect_ratio", "pad")
335
+ eval_logger.info(f"In Multi-Image setting, image aspect ratio: {self._config.image_aspect_ratio}")
336
+
337
+ if "task_type" in metadata and metadata["task_type"] == "video" and "sample_frames" in metadata: # overwrite logic for video task with multiple static image frames
338
+ assert type(visual) == list, "sample_frames must be specified for video task"
339
+ sample_indices = np.linspace(0, len(visual) - 1, metadata["sample_frames"], dtype=int)
340
+ visual = [visual[i] for i in sample_indices]
341
+ assert len(visual) == metadata["sample_frames"]
342
+
343
+ image_tensor = process_images(visual, self._image_processor, self._config)
344
+ if type(image_tensor) is list:
345
+ image_tensor = [_image.to(dtype=torch.float16, device=self.device) for _image in image_tensor]
346
+ else:
347
+ image_tensor = image_tensor.to(dtype=torch.float16, device=self.device)
348
+ image_tensor = [image_tensor]
349
+ task_type = "video"
350
+ placeholder_count = 1
351
+
352
+ elif type(visual[0]) == PIL.Image.Image: # For image, multi-image tasks
353
+ image_tensor = process_images(visual, self._image_processor, self._config)
354
+ speech = torch.zeros(3000, 128)
355
+ speech_lengths = torch.LongTensor([3000])
356
+ if type(image_tensor) is list:
357
+ image_tensor = [_image.to(dtype=torch.float16, device=self.device) for _image in image_tensor]
358
+ else:
359
+ image_tensor = image_tensor.to(dtype=torch.float16, device=self.device)
360
+
361
+ task_type = "image"
362
+ placeholder_count = len(visual) if isinstance(visual, list) else 1
363
+
364
+ elif type(visual[0]) == str: # For video task
365
+ image_tensor = []
366
+ try:
367
+ if self.video_decode_backend == "decord":
368
+ if "egoplan" in visual[0]:
369
+ task_name = "egoplan"
370
+ else:
371
+ task_name = None
372
+ frames, speech, speech_lengths = self.load_video(video_path=visual[0], max_frames_num=self.max_frames_num, task_name=task_name)
373
+ else:
374
+ raise NotImplementedError("Only decord backend is supported for video task")
375
+ processed_frames = self._image_processor.preprocess(frames, return_tensors="pt")["pixel_values"].half().cuda()
376
+ processed_frames = processed_frames.half()
377
+ image_tensor.append(processed_frames)
378
+ image_sizes = [frames[0].size]
379
+ except Exception as e:
380
+ eval_logger.error(f"Error {e} in loading video")
381
+ image_tensor = None
382
+
383
+ task_type = "video"
384
+ placeholder_count = len(frames) if self.token_strategy == "multiple" else 1
385
+ if DEFAULT_IMAGE_TOKEN not in context:
386
+ question = DEFAULT_IMAGE_TOKEN + "\n" + context
387
+ else:
388
+ question = context
389
+ speech = torch.stack([speech]).to(self.device).half()
390
+ # This is much safer for llama3, as we now have some object type in it
391
+ if "llama_3" in self.conv_template:
392
+ conv = copy.deepcopy(conv_templates[self.conv_template])
393
+ else:
394
+ conv = conv_templates[self.conv_template].copy()
395
+
396
+ if utils.is_json(question): # conversational question input
397
+ question = json.loads(question)
398
+ for idx, item in enumerate(question):
399
+ role = conv.roles[idx % 2]
400
+ message = item["value"]
401
+ conv.append_message(role, message)
402
+
403
+ assert len(conv.messages) % 2 == 1
404
+ conv.append_message(conv.roles[1], None)
405
+ prompt_question = conv.get_prompt()
406
+ question_input.append(prompt_question)
407
+ else: # only simple string for question
408
+ conv.append_message(conv.roles[0], question)
409
+ conv.append_message(conv.roles[1], None)
410
+ prompt_question = conv.get_prompt()
411
+ question_input.append(prompt_question)
412
+
413
+ # preconfigure gen_kwargs with defaults
414
+ if "max_new_tokens" not in gen_kwargs:
415
+ gen_kwargs["max_new_tokens"] = 1024
416
+ if "temperature" not in gen_kwargs:
417
+ gen_kwargs["temperature"] = 0
418
+ if "do_sample" not in gen_kwargs:
419
+ gen_kwargs["do_sample"] = False
420
+ if "top_p" not in gen_kwargs:
421
+ gen_kwargs["top_p"] = None
422
+ if "num_beams" not in gen_kwargs:
423
+ gen_kwargs["num_beams"] = 1
424
+
425
+ parts = self.split_text(prompt_question, ["<image>", "<speech>"])
426
+ input_ids = []
427
+ for part in parts:
428
+ if "<image>" == part:
429
+ input_ids += [IMAGE_TOKEN_INDEX]
430
+ elif "<speech>" == part:
431
+ input_ids += [SPEECH_TOKEN_INDEX]
432
+ else:
433
+ input_ids += self.tokenizer(part).input_ids
434
+
435
+ input_ids = torch.tensor(input_ids, dtype=torch.long).unsqueeze(0).to(self.device)
436
+ input_ids_list = [input_ids]
437
+ pad_token_ids = self.tokenizer.pad_token_id if self.tokenizer.pad_token_id is not None else self.tokenizer.eos_token_id
438
+ input_ids = self.pad_sequence(input_ids_list, batch_first=True, padding_value=pad_token_ids).to(self.device)
439
+ attention_masks = input_ids.ne(pad_token_ids).to(self.device)
440
+ input_ids = torch.tensor(input_ids, dtype=torch.long).squeeze(0).to(self.device)
441
+ if task_type == "image":
442
+ gen_kwargs["image_sizes"] = [batched_visuals[0][idx].size for idx in range(len(batched_visuals[0]))]
443
+ elif task_type == "video":
444
+ gen_kwargs["modalities"] = ["video"]
445
+ self._config.mm_spatial_pool_stride = self.mm_spatial_pool_stride
446
+ self._config.mm_spatial_pool_mode = self.mm_spatial_pool_mode
447
+ gen_kwargs["eos_token_id"] = self.tokenizer.eos_token_id
448
+
449
+ # These steps are not in LLaVA's original code, but are necessary for generation to work
450
+ # TODO: attention to this major generation step...
451
+ if "image_aspect_ratio" in gen_kwargs.keys():
452
+ gen_kwargs.pop("image_aspect_ratio")
453
+ try:
454
+ with torch.inference_mode():
455
+ cont = self.model.generate(input_ids, images=image_tensor, speech=speech, speech_lengths=speech_lengths, **gen_kwargs)
456
+
457
+ text_outputs = self.tokenizer.batch_decode(cont, skip_special_tokens=True)
458
+ except Exception as e:
459
+ raise e
460
+
461
+ text_outputs = [response.strip() for response in text_outputs]
462
+ res.extend(text_outputs)
463
+ self.cache_hook.add_partial("generate_until", (context, gen_kwargs), text_outputs)
464
+ pbar.update(1)
465
+ # reorder this group of results back to original unsorted form
466
+ res = re_ords.get_original(res)
467
+
468
+ pbar.close()
469
+ return res
470
+
471
+ def generate_until_multi_round(self, requests: List[Instance]) -> List[str]:
472
+ raise NotImplementedError("generate_until_multi_round is not implemented for EgoGPT")
LaViDa/eval/lmms_eval/models/minicpm_v.py ADDED
@@ -0,0 +1,224 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import warnings
2
+ from typing import List, Optional, Tuple, Union
3
+
4
+ import torch
5
+ from accelerate import Accelerator, DistributedType
6
+ from accelerate.state import AcceleratorState
7
+ from tqdm import tqdm
8
+ from transformers import AutoModel, AutoTokenizer
9
+
10
+ from lmms_eval import utils
11
+ from lmms_eval.api.instance import Instance
12
+ from lmms_eval.api.model import lmms
13
+ from lmms_eval.api.registry import register_model
14
+
15
+ warnings.filterwarnings("ignore")
16
+
17
+ from loguru import logger as eval_logger
18
+
19
+
20
+ @register_model("minicpm_v")
21
+ class MiniCPM_V(lmms):
22
+ """
23
+ MiniCPM_V Model
24
+ """
25
+
26
+ def __init__(
27
+ self,
28
+ pretrained: str = "openbmb/MiniCPM-V",
29
+ device: Optional[str] = "cuda",
30
+ dtype: Optional[Union[str, torch.dtype]] = torch.bfloat16,
31
+ batch_size: Optional[Union[int, str]] = 1,
32
+ trust_remote_code: Optional[bool] = True,
33
+ **kwargs,
34
+ ) -> None:
35
+ super().__init__()
36
+ # Do not use kwargs for now
37
+ assert kwargs == {}, f"Unexpected kwargs: {kwargs}"
38
+
39
+ accelerator = Accelerator()
40
+ if accelerator.num_processes > 1:
41
+ self._device = torch.device(f"cuda:{accelerator.local_process_index}")
42
+ else:
43
+ self._device = device
44
+ self._model = AutoModel.from_pretrained(pretrained, trust_remote_code=trust_remote_code, torch_dtype=dtype, device_map=self._device).to(dtype)
45
+ self._tokenizer = AutoTokenizer.from_pretrained(pretrained, trust_remote_code=trust_remote_code)
46
+ self._config = self._model.config
47
+ self.model.eval()
48
+ self.model.tie_weights()
49
+ self.batch_size_per_gpu = int(batch_size)
50
+ if accelerator.num_processes > 1:
51
+ assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported."
52
+ # If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model
53
+ # Also, you have to select zero stage 0 (equivalent to DDP) in order to make the prepare model works
54
+ # I tried to set different parameters in the kwargs to let default zero 2 stage works, but it didn't work.
55
+ if accelerator.distributed_type == DistributedType.DEEPSPEED:
56
+ kwargs = {
57
+ "train_micro_batch_size_per_gpu": self.batch_size_per_gpu,
58
+ "train_batch_size": self.batch_size_per_gpu * accelerator.num_processes,
59
+ }
60
+ AcceleratorState().deepspeed_plugin.deepspeed_config_process(must_match=True, **kwargs)
61
+ eval_logger.info("Detected that you are using DistributedType.DEEPSPEED. Make sure you run `accelerate config` and set zero stage to 0")
62
+ if accelerator.distributed_type == DistributedType.FSDP or accelerator.distributed_type == DistributedType.DEEPSPEED:
63
+ self._model = accelerator.prepare(self.model)
64
+ else:
65
+ self._model = accelerator.prepare_model(self.model, evaluation_mode=True)
66
+ self.accelerator = accelerator
67
+ if self.accelerator.is_local_main_process:
68
+ eval_logger.info(f"Using {accelerator.num_processes} devices with data parallelism")
69
+ self._rank = self.accelerator.local_process_index
70
+ self._world_size = self.accelerator.num_processes
71
+ else:
72
+ self.model.to(self._device)
73
+ self._rank = 0
74
+ self._world_size = 1
75
+
76
+ @property
77
+ def config(self):
78
+ # return the associated transformers.AutoConfig for the given pretrained model.
79
+ return self._config
80
+
81
+ @property
82
+ def tokenizer(self):
83
+ return self._tokenizer
84
+
85
+ @property
86
+ def model(self):
87
+ # returns the model, unwrapping it if using Accelerate
88
+ if hasattr(self, "accelerator"):
89
+ return self.accelerator.unwrap_model(self._model)
90
+ else:
91
+ return self._model
92
+
93
+ @property
94
+ def eot_token_id(self):
95
+ # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
96
+ return self.tokenizer.eos_token_id
97
+
98
+ @property
99
+ def max_length(self):
100
+ return self._max_length
101
+
102
+ @property
103
+ def batch_size(self):
104
+ return self.batch_size_per_gpu
105
+
106
+ @property
107
+ def device(self):
108
+ return self._device
109
+
110
+ @property
111
+ def rank(self):
112
+ return self._rank
113
+
114
+ @property
115
+ def world_size(self):
116
+ return self._world_size
117
+
118
+ def tok_encode(self, string: str, left_truncate_len=None, add_special_tokens=None) -> List[int]:
119
+ """ """
120
+ add_special_tokens = False if add_special_tokens is None else add_special_tokens
121
+ encoding = self.tokenizer.encode(string, add_special_tokens=add_special_tokens)
122
+ # left-truncate the encoded context to be at most `left_truncate_len` tokens long
123
+ if left_truncate_len:
124
+ encoding = encoding[-left_truncate_len:]
125
+ return encoding
126
+
127
+ def tok_decode(self, tokens):
128
+ return self.tokenizer.decode(tokens)
129
+
130
+ def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
131
+ # TODO
132
+ assert False, "We have not implemented this function for MiniCPM_V yet"
133
+
134
+ def flatten(self, input):
135
+ new_list = []
136
+ for i in input:
137
+ for j in i:
138
+ new_list.append(j)
139
+ return new_list
140
+
141
+ def generate_until(self, requests: List[Instance]) -> List[str]:
142
+ res = []
143
+
144
+ def _collate(x):
145
+ # the negative sign on len(toks) sorts descending - this has a few advantages:
146
+ # - time estimates will always be over not underestimates, which is more useful for planning
147
+ # - to know the size of a batch when going through the list, you know the first one is always the batch
148
+ # padded context length. this is useful to simplify the batching logic and more importantly to make
149
+ # automatic adaptive batches much much easier to implement
150
+ # - any OOMs will happen right away rather than near the end
151
+ toks = self.tok_encode(x[0])
152
+ return -len(toks), x[0]
153
+
154
+ # we group requests by their generation_kwargs,
155
+ # so that we don't try to execute e.g. greedy sampling and temp=0.8 sampling
156
+ # in the same batch.
157
+ re_ords = utils.Collator([reg.args for reg in requests], _collate, grouping=True)
158
+ chunks = re_ords.get_batched(n=self.batch_size, batch_fn=None)
159
+ num_iters = len(requests) // self.batch_size if len(requests) % self.batch_size == 0 else len(requests) // self.batch_size + 1
160
+ pbar = tqdm(total=num_iters, disable=(self.rank != 0), desc="Model Responding")
161
+ for chunk in chunks:
162
+ contexts, all_gen_kwargs, doc_to_visual, doc_id, task, split = zip(*chunk)
163
+ task = task[0]
164
+ split = split[0]
165
+ visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]
166
+ visuals = self.flatten(visuals)
167
+ # we assume all gen kwargs in the batch are the same
168
+ # this is safe to assume because the `grouper` object ensures it.
169
+ gen_kwargs = all_gen_kwargs[0]
170
+
171
+ # Set default values for until and max_new_tokens
172
+ until = [self.tok_decode(self.eot_token_id)]
173
+
174
+ # Update values from gen_kwargs if present
175
+ if "until" in gen_kwargs:
176
+ until = gen_kwargs.pop("until")
177
+ if isinstance(until, str):
178
+ until = [until]
179
+ elif not isinstance(until, list):
180
+ raise ValueError(f"Expected `gen_kwargs['until']` to be of type Union[str,list] but got {type(until)}")
181
+ assert self.batch_size_per_gpu == 1, "Do not support batch_size_per_gpu > 1 for now"
182
+ assert len(visuals) == 1, "MiniCPM_V interface does not support bn_image > 1 for now"
183
+ context = contexts[0]
184
+ if "<image>" in context:
185
+ # minicpm does not expect the <image> tag
186
+ context = context.replace("<image>", "")
187
+ msgs = [{"role": "user", "content": context}]
188
+
189
+ gen_kwargs["image_sizes"] = [visuals[idx].size for idx in range(len(visuals))]
190
+ if "max_new_tokens" not in gen_kwargs:
191
+ gen_kwargs["max_new_tokens"] = 1024
192
+ if "temperature" not in gen_kwargs:
193
+ gen_kwargs["temperature"] = 0
194
+ if "top_p" not in gen_kwargs:
195
+ gen_kwargs["top_p"] = None
196
+ if "num_beams" not in gen_kwargs:
197
+ gen_kwargs["num_beams"] = 1
198
+ try:
199
+ # ominicpm does not give much information on how they do eval so I just use the chat format.
200
+ response, context, _ = self.model.chat(
201
+ image=visuals[0],
202
+ msgs=msgs,
203
+ context=None,
204
+ tokenizer=self.tokenizer,
205
+ sampling=True if gen_kwargs["temperature"] > 0 else False,
206
+ temperature=gen_kwargs["temperature"],
207
+ top_p=gen_kwargs["top_p"],
208
+ num_beams=gen_kwargs["num_beams"],
209
+ max_new_tokens=gen_kwargs["max_new_tokens"],
210
+ )
211
+ except Exception as e:
212
+ eval_logger.error(f"Error {e} in generating")
213
+ cont = ""
214
+ res.append(response)
215
+ self.cache_hook.add_partial("generate_until", (context, gen_kwargs), response)
216
+ pbar.update(1)
217
+ # reorder this group of results back to original unsorted form
218
+ res = re_ords.get_original(res)
219
+
220
+ pbar.close()
221
+ return res
222
+
223
+ def generate_until_multi_round(self, requests) -> List[str]:
224
+ raise NotImplementedError("TODO: Implement multi-round generation")
LaViDa/eval/lmms_eval/models/moviechat.py ADDED
@@ -0,0 +1,452 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import copy
2
+ import json
3
+ import logging
4
+ import math
5
+ import os
6
+ import os.path as osp
7
+ import queue
8
+ import re
9
+ import warnings
10
+ from datetime import timedelta
11
+ from typing import List, Optional, Tuple, Union
12
+
13
+ import av
14
+ import einops
15
+ import numpy as np
16
+ import PIL
17
+ import torch
18
+ import transformers
19
+ from accelerate import Accelerator, DistributedType, InitProcessGroupKwargs
20
+ from accelerate.state import AcceleratorState
21
+ from decord import VideoReader, cpu
22
+ from huggingface_hub import snapshot_download
23
+ from moviepy.video.io.VideoFileClip import VideoFileClip
24
+ from packaging import version
25
+ from PIL import Image
26
+ from scipy.spatial.distance import cosine
27
+ from skimage import transform
28
+ from torchvision import transforms
29
+ from torchvision.transforms.functional import InterpolationMode
30
+ from tqdm import tqdm
31
+ from transformers import StoppingCriteria, StoppingCriteriaList
32
+
33
+ from lmms_eval import utils
34
+ from lmms_eval.api.instance import Instance
35
+ from lmms_eval.api.model import lmms
36
+ from lmms_eval.api.registry import register_model
37
+
38
+ # Suppress warnings
39
+ warnings.filterwarnings("ignore")
40
+
41
+ # Configure logging
42
+ eval_logger = logging.getLogger("lmms-eval")
43
+
44
+ # Enable TF32 for CUDA
45
+ torch.backends.cuda.matmul.allow_tf32 = True
46
+
47
+ # Import LLaVA modules
48
+ try:
49
+ from MovieChat.common.registry import registry
50
+ except ImportError as e:
51
+ eval_logger.debug(
52
+ f"MovieChat is not installed. First, install MovieChat by 'https://github.com/rese1f/MovieChat.git' and 'cd MovieChat'. Change the torch version with `python -m pip install torch==2.0.1 torchvision==0.15.2 --extra-index-url https://download.pytorch.org/whl/cu118`"
53
+ )
54
+
55
+
56
+ # Determine best attention implementation
57
+ if version.parse(torch.__version__) >= version.parse("2.1.2"):
58
+ best_fit_attn_implementation = "sdpa"
59
+ else:
60
+ best_fit_attn_implementation = "eager"
61
+
62
+
63
+ class StoppingCriteriaSub(StoppingCriteria):
64
+ def __init__(self, stops=[], encounters=1):
65
+ super().__init__()
66
+ self.stops = stops
67
+
68
+ def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor):
69
+ for stop in self.stops:
70
+ if torch.all((stop == input_ids[0][-len(stop) :])).item():
71
+ return True
72
+
73
+ return False
74
+
75
+
76
+ @register_model("moviechat")
77
+ class MovieChat(lmms):
78
+ """
79
+ MovieChat Model
80
+ """
81
+
82
+ def __init__(
83
+ self,
84
+ truncation: Optional[bool] = True,
85
+ device: Optional[str] = "cuda:0",
86
+ batch_size: Optional[Union[int, str]] = 1,
87
+ pretrained_llama_model: str = "Enxin/MovieChat-vicuna",
88
+ pretrained_llama_proj_model: str = "Enxin/MovieChat-proj",
89
+ attn_implementation: Optional[str] = best_fit_attn_implementation,
90
+ device_map: Optional[str] = "cuda:0",
91
+ use_cache: Optional[bool] = True,
92
+ truncate_context: Optional[bool] = False, # whether to truncate the context in generation, set it False for LLaVA-1.6
93
+ customized_config: Optional[str] = None, # ends in json
94
+ short_memory_length: Optional[int] = 18,
95
+ long_memory_length: Optional[int] = 256,
96
+ sliding_window_length: Optional[int] = 8,
97
+ merge_frame_length: Optional[int] = 2,
98
+ tmp_folder: Optional[str] = "tmp/",
99
+ **kwargs,
100
+ ) -> None:
101
+ super().__init__()
102
+ # Do not use kwargs for now
103
+ assert kwargs == {}, f"Unexpected kwargs: {kwargs}"
104
+
105
+ accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52))
106
+ accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs])
107
+ if accelerator.num_processes > 1:
108
+ self._device = torch.device(f"cuda:{accelerator.local_process_index}")
109
+ self.device_map = f"cuda:{accelerator.local_process_index}"
110
+ elif accelerator.num_processes == 1 and device_map == "auto":
111
+ self._device = torch.device(device)
112
+ self.device_map = device_map
113
+ else:
114
+ self._device = torch.device(f"cuda:{accelerator.local_process_index}")
115
+ self.device_map = f"cuda:{accelerator.local_process_index}"
116
+
117
+ llama_model = snapshot_download(repo_id=pretrained_llama_model) if not osp.isdir(pretrained_llama_model) else pretrained_llama_model
118
+ llama_proj_pth = snapshot_download(repo_id=pretrained_llama_proj_model) if not osp.isdir(pretrained_llama_proj_model) else pretrained_llama_proj_model
119
+ llama_proj = osp.join(llama_proj_pth, "finetune-vicuna7b-v2.pth")
120
+ model_config = {
121
+ "arch": "moviechat",
122
+ "model_type": "pretrain_vicuna",
123
+ "freeze_vit": True,
124
+ "freeze_qformer": True,
125
+ "max_txt_len": 256,
126
+ "end_sym": "###",
127
+ "low_resource": False,
128
+ "frozen_llama_proj": False,
129
+ "llama_model": llama_model,
130
+ "llama_proj_model": llama_proj,
131
+ }
132
+
133
+ model_cls = registry.get_model_class(model_config["arch"])
134
+ self._model = model_cls.from_config(model_config).to(self.device_map)
135
+
136
+ vis_processor_cfg = {
137
+ "name": "alpro_video_eval",
138
+ "n_frms": 8,
139
+ }
140
+ self.transform = transforms.Compose(
141
+ [transforms.Resize((224, 224)), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])] # Resize to 224x224 # Convert PIL Image to Tensor with shape [C, H, W] # Normalize
142
+ )
143
+ self._image_processor = registry.get_processor_class(vis_processor_cfg["name"]).from_config(vis_processor_cfg)
144
+
145
+ self.model.short_memory_length = short_memory_length
146
+ self.model.long_memory_length = long_memory_length
147
+ self.merge_frame_length = merge_frame_length
148
+ self.sliding_window_length = sliding_window_length
149
+ self.num_clips = (self.model.long_memory_length // self.merge_frame_length) * ((self.model.short_memory_length - self.merge_frame_length) // self.sliding_window_length)
150
+ self.tmp_folder = tmp_folder
151
+
152
+ self._tokenizer = self.model.llama_tokenizer
153
+ stop_words_ids = [torch.tensor([835]).to(self.device), torch.tensor([2277, 29937]).to(self.device)] # '###' can be encoded in two different ways.
154
+ self.stopping_criteria = StoppingCriteriaList([StoppingCriteriaSub(stops=stop_words_ids)])
155
+
156
+ self.model.eval()
157
+ self.truncation = truncation
158
+ self.batch_size_per_gpu = int(batch_size)
159
+ self.use_cache = use_cache
160
+ self.truncate_context = truncate_context
161
+ assert self.batch_size_per_gpu == 1, "MovieChat currently does not support batched generation."
162
+
163
+ if accelerator.num_processes > 1:
164
+ assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported."
165
+ # If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model
166
+ # Also, you have to select zero stage 0 (equivalent to DDP) in order to make the prepare model works
167
+ # I tried to set different parameters in the kwargs to let default zero 2 stage works, but it didn't work.
168
+ if accelerator.distributed_type == DistributedType.DEEPSPEED:
169
+ kwargs = {
170
+ "train_micro_batch_size_per_gpu": self.batch_size_per_gpu,
171
+ "train_batch_size": self.batch_size_per_gpu * accelerator.num_processes,
172
+ }
173
+ AcceleratorState().deepspeed_plugin.deepspeed_config_process(must_match=True, **kwargs)
174
+ eval_logger.info("Detected that you are using DistributedType.DEEPSPEED. Make sure you run `accelerate config` and set zero stage to 0")
175
+
176
+ if accelerator.distributed_type == DistributedType.FSDP or accelerator.distributed_type == DistributedType.DEEPSPEED:
177
+ self._model = accelerator.prepare(self.model)
178
+ else:
179
+ self._model = accelerator.prepare_model(self.model, evaluation_mode=True)
180
+ self.accelerator = accelerator
181
+ if self.accelerator.is_local_main_process:
182
+ eval_logger.info(f"Using {accelerator.num_processes} devices with data parallelism")
183
+ self._rank = self.accelerator.local_process_index
184
+ self._world_size = self.accelerator.num_processes
185
+
186
+ elif accelerator.num_processes == 1 and device_map == "auto":
187
+ eval_logger.info(f"Using {accelerator.num_processes} devices with tensor parallelism")
188
+ self._rank = 0
189
+ self._world_size = 1
190
+
191
+ else:
192
+ eval_logger.info(f"Using single device: {self._device}")
193
+ self.model.to(self._device)
194
+ self._rank = 0
195
+ self._world_size = 1
196
+
197
+ @property
198
+ def config(self):
199
+ # return the associated transformers.AutoConfig for the given pretrained model.
200
+ return self._config
201
+
202
+ @property
203
+ def tokenizer(self):
204
+ return self._tokenizer
205
+
206
+ @property
207
+ def model(self):
208
+ # returns the model, unwrapping it if using Accelerate
209
+ if hasattr(self, "accelerator"):
210
+ return self.accelerator.unwrap_model(self._model)
211
+ else:
212
+ return self._model
213
+
214
+ @property
215
+ def eot_token_id(self):
216
+ # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
217
+ return self.tokenizer.eos_token_id
218
+
219
+ @property
220
+ def max_length(self):
221
+ return self._max_length
222
+
223
+ @property
224
+ def batch_size(self):
225
+ return self.batch_size_per_gpu
226
+
227
+ @property
228
+ def device(self):
229
+ return self._device
230
+
231
+ @property
232
+ def rank(self):
233
+ return self._rank
234
+
235
+ @property
236
+ def world_size(self):
237
+ return self._world_size
238
+
239
+ def tok_encode(self, string: str, left_truncate_len=None, add_special_tokens=None) -> List[int]:
240
+ """ """
241
+ add_special_tokens = False if add_special_tokens is None else add_special_tokens
242
+ encoding = self.tokenizer.encode(string, add_special_tokens=add_special_tokens)
243
+ # left-truncate the encoded context to be at most `left_truncate_len` tokens long
244
+ if left_truncate_len:
245
+ encoding = encoding[-left_truncate_len:]
246
+ return encoding
247
+
248
+ def tok_decode(self, tokens):
249
+ try:
250
+ return self.tokenizer.decode(tokens)
251
+ except:
252
+ return self.tokenizer.decode([tokens])
253
+
254
+ def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
255
+ # TODO
256
+ raise NotImplementedError("MovieChat only supports generation.")
257
+
258
+ def flatten(self, input):
259
+ new_list = []
260
+ for i in input:
261
+ for j in i:
262
+ new_list.append(j)
263
+ return new_list
264
+
265
+ def get_context_emb(self, input_text, img_list):
266
+ prompt_1 = "You are able to understand the visual content that the user provides.Follow the instructions carefully and explain your answers in details.###Human: <Video><ImageHere></Video>"
267
+ prompt_2 = input_text
268
+ prompt_3 = "###Assistant:"
269
+
270
+ prompt = prompt_1 + " " + prompt_2 + prompt_3
271
+
272
+ prompt_segs = prompt.split("<ImageHere>")
273
+ assert len(prompt_segs) == len(img_list) + 1, "Unmatched numbers of image placeholders and images."
274
+ seg_tokens = [
275
+ self.model.llama_tokenizer(seg, return_tensors="pt", add_special_tokens=i == 0).to(self.device).input_ids
276
+ # only add bos to the first seg
277
+ for i, seg in enumerate(prompt_segs)
278
+ ]
279
+ seg_embs = [self.model.llama_model.model.embed_tokens(seg_t) for seg_t in seg_tokens]
280
+
281
+ mixed_embs = [emb for pair in zip(seg_embs[:-1], img_list) for emb in pair] + [seg_embs[-1]]
282
+ mixed_embs = torch.cat(mixed_embs, dim=1)
283
+ return mixed_embs
284
+
285
+ def answer(self, img_list, input_text, max_new_tokens=300, num_beams=1, min_length=1, top_p=0.9, repetition_penalty=1.0, length_penalty=1, temperature=1.0, max_length=2000):
286
+ embs = self.get_context_emb(input_text, img_list)
287
+
288
+ current_max_len = embs.shape[1] + max_new_tokens
289
+ if current_max_len - max_length > 0:
290
+ print("Warning: The number of tokens in current conversation exceeds the max length. " "The model will not see the contexts outside the range.")
291
+ begin_idx = max(0, current_max_len - max_length)
292
+
293
+ embs = embs[:, begin_idx:]
294
+
295
+ outputs = self.model.llama_model.generate(
296
+ inputs_embeds=embs,
297
+ max_new_tokens=max_new_tokens,
298
+ stopping_criteria=self.stopping_criteria,
299
+ num_beams=num_beams,
300
+ do_sample=True,
301
+ min_length=min_length,
302
+ top_p=top_p,
303
+ repetition_penalty=repetition_penalty,
304
+ length_penalty=length_penalty,
305
+ temperature=temperature,
306
+ )
307
+
308
+ output_token = outputs[0]
309
+ if output_token[0] == 0: # the model might output a unknow token
310
+ output_token = output_token[1:]
311
+ if output_token[0] == 1: # some users find that there is a start token at the beginning. remove it
312
+ output_token = output_token[1:]
313
+ output_text = self.model.llama_tokenizer.decode(output_token, add_special_tokens=False)
314
+ output_text = output_text.split("###")[0] # remove the stop sign '###'
315
+ output_text = output_text.split("Assistant:")[-1].strip()
316
+ return output_text, output_token.cpu().numpy()
317
+
318
+ def generate_until(self, requests: List[Instance]) -> List[str]:
319
+ res = []
320
+
321
+ def _collate(x):
322
+ # the negative sign on len(toks) sorts descending - this has a few advantages:
323
+ # - time estimates will always be over not underestimates, which is more useful for planning
324
+ # - to know the size of a batch when going through the list, you know the first one is always the batch
325
+ # padded context length. this is useful to simplify the batching logic and more importantly to make
326
+ # automatic adaptive batches much much easier to implement
327
+ # - any OOMs will happen right away rather than near the end
328
+ toks = self.tok_encode(x[0])
329
+ return -len(toks), x[0]
330
+
331
+ # we group requests by their generation_kwargs,
332
+ # so that we don't try to execute e.g. greedy sampling and temp=0.8 sampling
333
+ # in the same batch.
334
+ metadata = requests[0].metadata
335
+ re_ords = utils.Collator([reg.args for reg in requests], _collate, grouping=True)
336
+ chunks = re_ords.get_batched(n=self.batch_size, batch_fn=None)
337
+ num_iters = len(requests) // self.batch_size if len(requests) % self.batch_size == 0 else len(requests) // self.batch_size + 1
338
+ pbar = tqdm(total=num_iters, disable=(self.rank != 0), desc="Model Responding")
339
+ for chunk in chunks:
340
+ batched_contexts, all_gen_kwargs, batched_doc_to_visual, batched_doc_id, batched_task, batched_split = zip(*chunk)
341
+ task = batched_task[0]
342
+ split = batched_split[0]
343
+ batched_visuals = [batched_doc_to_visual[0](self.task_dict[task][split][ids]) for ids in batched_doc_id] # [B, N]
344
+ assert len(batched_visuals) == 1
345
+
346
+ # we assume all gen kwargs in the batch are the same
347
+ # this is safe to assume because the `grouper` object ensures it.
348
+ gen_kwargs = all_gen_kwargs[0]
349
+ if "until" in gen_kwargs:
350
+ gen_kwargs.pop("until")
351
+
352
+ text_outputs = []
353
+
354
+ for visual, context in zip(batched_visuals, batched_contexts):
355
+ if type(visual[0]) == PIL.Image.Image and "task_type" not in metadata and "sample_frames" not in metadata: # For image task
356
+ raise NotImplementedError("MovieChat only supports video inputs.")
357
+
358
+ elif "task_type" in metadata and metadata["task_type"] == "video" and "sample_frames" in metadata:
359
+ raise NotImplementedError("MovieChat only supports video inputs.")
360
+
361
+ elif type(visual[0]) == str: # For video task
362
+ image_tensor = []
363
+ self.model.short_memory_buffer = []
364
+ self.model.long_memory_buffer = []
365
+ img_list = []
366
+ # try:
367
+ os.makedirs(self.tmp_folder, exist_ok=True)
368
+
369
+ video = VideoFileClip(visual[0])
370
+ clip_duration = video.duration / self.num_clips
371
+
372
+ cur_frame = 0
373
+ for i in range(self.num_clips):
374
+ preprocess_frames = []
375
+ start_time = i * clip_duration
376
+ end_time = start_time + clip_duration
377
+ # uniformly sample self.sliding_window_length frames from the video from start_time to end_time
378
+ frames = list(video.subclip(start_time, end_time).iter_frames(fps=self.sliding_window_length / clip_duration))[: self.sliding_window_length]
379
+ for frame in frames:
380
+ frame = Image.fromarray(frame)
381
+ frame_tensor = self.transform(frame)
382
+ frame_tensor = frame_tensor.permute(2, 0, 1)
383
+ frame_tensor = frame_tensor.unsqueeze(0)
384
+ frame_tensor = self._image_processor.transform(frame_tensor)
385
+ frame_tensor = frame_tensor.squeeze(-1).permute(1, 2, 0)
386
+ preprocess_frames.append(frame_tensor)
387
+
388
+ frames_tensor = torch.stack(preprocess_frames, dim=0)
389
+
390
+ image_embeds = self.model.ln_vision(self.model.visual_encoder(frames_tensor.half().to(self.device)))
391
+ image_atts = torch.ones(image_embeds.size()[:-1], dtype=torch.long).to(self.device)
392
+ query_tokens = self.model.query_tokens.expand(image_embeds.shape[0], -1, -1)
393
+ query_output = self.model.Qformer.bert(
394
+ query_embeds=query_tokens,
395
+ encoder_hidden_states=image_embeds,
396
+ encoder_attention_mask=image_atts,
397
+ return_dict=True,
398
+ )
399
+ encoded_window = query_output.last_hidden_state
400
+
401
+ for frame in encoded_window:
402
+ if cur_frame < (self.model.short_memory_length - self.merge_frame_length):
403
+ if len(self.model.short_memory_buffer) == self.model.short_memory_length:
404
+ self.model.short_memory_buffer.pop(0)
405
+ self.model.short_memory_buffer.append(frame)
406
+ cur_frame += 1
407
+
408
+ if cur_frame == (self.model.short_memory_length - self.merge_frame_length):
409
+ cur_frame = 0
410
+
411
+ # merge short_memory_frames
412
+ similar_list = []
413
+ for frame_i in range(len(self.model.short_memory_buffer) - 1):
414
+ scores = self.model.short_memory_buffer[frame_i] @ self.model.short_memory_buffer[frame_i + 1].transpose(-1, -2)
415
+ frame_silimar = torch.mean(scores)
416
+ similar_list.append(frame_silimar)
417
+
418
+ while len(self.model.short_memory_buffer) > self.merge_frame_length:
419
+ max_value = max(similar_list)
420
+ max_index = similar_list.index(max_value)
421
+ new_frame_feature = (self.model.short_memory_buffer[max_index].cpu() + self.model.short_memory_buffer[max_index + 1].cpu()) / 2
422
+ self.model.short_memory_buffer[max_index] = new_frame_feature.cuda()
423
+ del self.model.short_memory_buffer[max_index + 1]
424
+ similar_list = []
425
+ for frame_i in range(len(self.model.short_memory_buffer) - 1):
426
+ scores = self.model.short_memory_buffer[frame_i] @ self.model.short_memory_buffer[frame_i + 1].transpose(-1, -2)
427
+ frame_silimar = torch.mean(scores)
428
+ similar_list.append(frame_silimar)
429
+
430
+ for frame in self.model.short_memory_buffer:
431
+ self.model.long_memory_buffer.append(frame)
432
+
433
+ cur_image = self.model.encode_image(preprocess_frames[-1].unsqueeze(0).unsqueeze(2).half(), self.device)
434
+ video_emb, _ = self.model.encode_long_video(cur_image, device=self.device, middle_video=False)
435
+ img_list.append(video_emb)
436
+ llm_message = self.answer(img_list=img_list, input_text=context, num_beams=1, temperature=1.0, max_new_tokens=300, max_length=2000)[0]
437
+ text_outputs.append(llm_message)
438
+
439
+ # except Exception as e:
440
+ # eval_logger.error(f"Error {e} in loading video")
441
+ # image_tensor = None
442
+
443
+ text_outputs = [response.strip() for response in text_outputs]
444
+ res.extend(text_outputs)
445
+ print(text_outputs)
446
+ self.cache_hook.add_partial("generate_until", (context, gen_kwargs), text_outputs)
447
+ pbar.update(1)
448
+ # reorder this group of results back to original unsorted form
449
+ res = re_ords.get_original(res)
450
+
451
+ pbar.close()
452
+ return res
LaViDa/eval/lmms_eval/models/video_llava.py ADDED
@@ -0,0 +1,214 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import math
2
+ from datetime import timedelta
3
+ from typing import List, Optional, Tuple, Union
4
+
5
+ import numpy as np
6
+ import torch
7
+ from accelerate import Accelerator, DistributedType, InitProcessGroupKwargs
8
+ from accelerate.state import AcceleratorState
9
+ from loguru import logger
10
+ from tqdm import tqdm
11
+ from transformers import AutoConfig
12
+
13
+ from lmms_eval import utils
14
+ from lmms_eval.api.instance import Instance
15
+ from lmms_eval.api.model import lmms
16
+ from lmms_eval.api.registry import register_model
17
+ from lmms_eval.utils import stop_sequences_criteria
18
+
19
+ eval_logger = logger
20
+
21
+ from transformers import VideoLlavaForConditionalGeneration, VideoLlavaProcessor
22
+
23
+ from lmms_eval.models.model_utils.load_video import read_video_pyav
24
+
25
+
26
+ @register_model("video_llava")
27
+ class VideoLLaVA(lmms):
28
+ def __init__(
29
+ self,
30
+ pretrained: str = "LanguageBind/Video-LLaVA-7B-hf",
31
+ truncation: Optional[bool] = True,
32
+ device: Optional[str] = "cuda:0",
33
+ dtype: Optional[Union[str, torch.dtype]] = "auto",
34
+ batch_size: Optional[Union[int, str]] = 1,
35
+ trust_remote_code: Optional[bool] = False,
36
+ revision=None,
37
+ attn_implementation=(
38
+ "sdpa" if torch.__version__ > "2.1.2" else "eager"
39
+ ), # inference implementation for attention, can be "sdpa", "eager", "flash_attention_2". Seems FA2 is not effective during inference: https://discuss.huggingface.co/t/flash-attention-has-no-effect-on-inference/73453/5
40
+ device_map="cuda:0",
41
+ conv_template="llava_v1",
42
+ use_cache=True,
43
+ truncate_context=False,
44
+ num_frames: int = 8, # whether to truncate the context in generation, set it False for LLaVA-1.6
45
+ **kwargs,
46
+ ) -> None:
47
+ super().__init__()
48
+ accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52))
49
+ accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs])
50
+ if accelerator.num_processes > 1:
51
+ self._device = torch.device(f"cuda:{accelerator.local_process_index}")
52
+ self.device_map = f"cuda:{accelerator.local_process_index}"
53
+ elif accelerator.num_processes == 1 and device_map == "auto":
54
+ self._device = torch.device(device)
55
+ self.device_map = device_map
56
+ else:
57
+ self._device = torch.device(f"cuda:{accelerator.local_process_index}")
58
+ self.device_map = f"cuda:{accelerator.local_process_index}"
59
+
60
+ self.pretrained = pretrained
61
+ self._model = VideoLlavaForConditionalGeneration.from_pretrained(pretrained)
62
+ self._processor = VideoLlavaProcessor.from_pretrained(pretrained)
63
+ self.prompt = "USER: <video>{}? ASSISTANT:"
64
+ self.num_frames = num_frames
65
+ assert num_frames == 8, "num_frames must be 8 https://github.com/huggingface/transformers/blob/bdb9106f247fca48a71eb384be25dbbd29b065a8/src/transformers/models/video_llava/modeling_video_llava.py#L379"
66
+ # self.model_name = get_model_name_from_path(pretrained)
67
+ # self._tokenizer, self._model, self.processor, self._max_length = load_pretrained_model(pretrained, None, self.model_name, device_map=self.device_map)
68
+ # self.video_processor = self.processor["video"]
69
+ self._config = self._model.config
70
+ self.model.eval()
71
+ self.model.tie_weights()
72
+ self.truncation = truncation
73
+ self.batch_size_per_gpu = int(batch_size)
74
+ self.conv_template = conv_template
75
+ self.use_cache = use_cache
76
+ self.truncate_context = truncate_context
77
+ # assert self.batch_size_per_gpu == 1, "Llava currently does not support batched generation. See https://github.com/haotian-liu/LLaVA/issues/754. HF Llava also has this issue."
78
+ if accelerator.num_processes > 1:
79
+ assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported."
80
+ # If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model
81
+ # Also, you have to select zero stage 0 (equivalent to DDP) in order to make the prepare model works
82
+ # I tried to set different parameters in the kwargs to let default zero 2 stage works, but it didn't work.
83
+ if accelerator.distributed_type == DistributedType.DEEPSPEED:
84
+ kwargs = {
85
+ "train_micro_batch_size_per_gpu": self.batch_size_per_gpu,
86
+ "train_batch_size": self.batch_size_per_gpu * accelerator.num_processes,
87
+ }
88
+ AcceleratorState().deepspeed_plugin.deepspeed_config_process(must_match=True, **kwargs)
89
+ eval_logger.info("Detected that you are using DistributedType.DEEPSPEED. Make sure you run `accelerate config` and set zero stage to 0")
90
+ if accelerator.distributed_type == DistributedType.FSDP or accelerator.distributed_type == DistributedType.DEEPSPEED:
91
+ self._model = accelerator.prepare(self.model)
92
+ else:
93
+ self._model = accelerator.prepare_model(self.model, evaluation_mode=True)
94
+ self.accelerator = accelerator
95
+ if self.accelerator.is_local_main_process:
96
+ eval_logger.info(f"Using {accelerator.num_processes} devices with data parallelism")
97
+ self._rank = self.accelerator.local_process_index
98
+ self._world_size = self.accelerator.num_processes
99
+ elif accelerator.num_processes == 1 and device_map == "auto":
100
+ eval_logger.info(f"Using {accelerator.num_processes} devices with tensor parallelism")
101
+ self._rank = 0
102
+ self._world_size = 1
103
+ else:
104
+ eval_logger.info(f"Using single device: {self._device}")
105
+ self.model.to(self._device)
106
+ self._rank = 0
107
+ self._world_size = 1
108
+
109
+ @property
110
+ def config(self):
111
+ # return the associated transformers.AutoConfig for the given pretrained model.
112
+ return self._config
113
+
114
+ @property
115
+ def tokenizer(self):
116
+ return self._tokenizer
117
+
118
+ @property
119
+ def model(self):
120
+ # returns the model, unwrapping it if using Accelerate
121
+ if hasattr(self, "accelerator"):
122
+ return self.accelerator.unwrap_model(self._model)
123
+ else:
124
+ return self._model
125
+
126
+ @property
127
+ def eot_token_id(self):
128
+ # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
129
+ return self.tokenizer.eos_token_id
130
+
131
+ @property
132
+ def max_length(self):
133
+ return self._max_length
134
+
135
+ def pad_sequence(self, input_ids, batch_first, padding_value):
136
+ if self.tokenizer.padding_side == "left":
137
+ input_ids = [torch.flip(_input_ids, [0]) for _input_ids in input_ids]
138
+ input_ids = torch.nn.utils.rnn.pad_sequence(input_ids, batch_first=batch_first, padding_value=padding_value)
139
+ if self.tokenizer.padding_side == "left":
140
+ input_ids = torch.flip(input_ids, [1])
141
+ return input_ids
142
+
143
+ @property
144
+ def batch_size(self):
145
+ return self.batch_size_per_gpu
146
+
147
+ @property
148
+ def device(self):
149
+ return self._device
150
+
151
+ @property
152
+ def rank(self):
153
+ return self._rank
154
+
155
+ @property
156
+ def world_size(self):
157
+ return self._world_size
158
+
159
+ def tok_encode(self, string: str, left_truncate_len=None, add_special_tokens=None) -> List[int]:
160
+ """ """
161
+ add_special_tokens = False if add_special_tokens is None else add_special_tokens
162
+ encoding = self.tokenizer.encode(string, add_special_tokens=add_special_tokens)
163
+ # left-truncate the encoded context to be at most `left_truncate_len` tokens long
164
+ if left_truncate_len:
165
+ encoding = encoding[-left_truncate_len:]
166
+ return encoding
167
+
168
+ def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
169
+ return super().loglikelihood(requests)
170
+
171
+ def flatten(self, input):
172
+ new_list = []
173
+ for i in input:
174
+ for j in i:
175
+ new_list.append(j)
176
+ return new_list
177
+
178
+ def generate_until(self, requests) -> List[str]:
179
+ res = []
180
+ pbar = tqdm(total=len(requests), disable=(self.rank != 0), desc="Model Responding")
181
+
182
+ for contexts, gen_kwargs, doc_to_visual, doc_id, task, split in [reg.args for reg in requests]:
183
+ # encode, pad, and truncate contexts for this batch
184
+ visuals = [doc_to_visual(self.task_dict[task][split][doc_id])]
185
+ visuals = self.flatten(visuals)
186
+ assert len(visuals) == 1
187
+ clip = read_video_pyav(visuals[0], self.num_frames)
188
+
189
+ inputs = self._processor(text=self.prompt.format(contexts), videos=clip, return_tensors="pt")
190
+ pixel_values_videos = inputs["pixel_values_videos"]
191
+ if pixel_values_videos.shape[1] != self.num_frames:
192
+ empty_frames = torch.zeros((1, self.num_frames - pixel_values_videos.shape[1], *pixel_values_videos.shape[2:]), dtype=pixel_values_videos.dtype)
193
+ pixel_values_videos = torch.cat([pixel_values_videos, empty_frames], dim=1)
194
+ inputs["pixel_values_videos"] = pixel_values_videos
195
+ inputs = {k: v.to(self.device) for k, v in inputs.items()}
196
+
197
+ if "max_new_tokens" not in gen_kwargs:
198
+ gen_kwargs["max_new_tokens"] = 1024
199
+ if "temperature" not in gen_kwargs:
200
+ gen_kwargs["temperature"] = 0
201
+ if "top_p" not in gen_kwargs:
202
+ gen_kwargs["top_p"] = None
203
+ if "num_beams" not in gen_kwargs:
204
+ gen_kwargs["num_beams"] = 1
205
+
206
+ generate_ids = self.model.generate(**inputs, max_new_tokens=gen_kwargs["max_new_tokens"], temperature=gen_kwargs["temperature"])
207
+
208
+ outputs = self._processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0].split("ASSISTANT:")[-1].strip()
209
+ res.append(outputs)
210
+ pbar.update(1)
211
+ return res
212
+
213
+ def generate_until_multi_round(self, requests) -> List[str]:
214
+ raise NotImplementedError("TODO: Implement multi-round generation")
LaViDa/eval/lmms_eval/tasks/activitynetqa/utils.py ADDED
@@ -0,0 +1,287 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import ast
2
+ import datetime
3
+ import json
4
+ import os
5
+ import sys
6
+ import time
7
+ from pathlib import Path
8
+
9
+ import numpy as np
10
+ import openai
11
+ import requests
12
+ import yaml
13
+ from decord import VideoReader, cpu
14
+ from loguru import logger as eval_logger
15
+ from openai import OpenAI
16
+
17
+ import lmms_eval.tasks._task_utils.file_utils as file_utils
18
+
19
+ with open(Path(__file__).parent / "_default_template_yaml", "r") as f:
20
+ raw_data = f.readlines()
21
+ safe_data = []
22
+ for i, line in enumerate(raw_data):
23
+ # remove function definition since yaml load cannot handle it
24
+ if "!function" not in line:
25
+ safe_data.append(line)
26
+
27
+ config = yaml.safe_load("".join(safe_data))
28
+
29
+ NUM_SECONDS_TO_SLEEP = 5
30
+
31
+ GPT_EVAL_MODEL_NAME = config["metadata"]["gpt_eval_model_name"]
32
+
33
+ API_TYPE = os.getenv("API_TYPE", "openai")
34
+
35
+ if API_TYPE == "openai":
36
+ API_URL = os.getenv("OPENAI_API_URL", "https://api.openai.com/v1/chat/completions")
37
+ API_KEY = os.getenv("OPENAI_API_KEY", "YOUR_API_KEY")
38
+ headers = {
39
+ "Authorization": f"Bearer {API_KEY}",
40
+ "Content-Type": "application/json",
41
+ }
42
+
43
+ # Unzip all the zip files to HF HOME cache dir
44
+ HF_HOME = os.environ["HF_HOME"]
45
+ cache_dir = config["dataset_kwargs"]["cache_dir"]
46
+ cache_dir = os.path.join(HF_HOME, cache_dir)
47
+ cache_dir = os.path.join(cache_dir, "all_test")
48
+
49
+
50
+ # Pass in video path here
51
+ # Can only work correctly with video dataset
52
+ def activitynetqa_doc_to_visual(doc):
53
+ video_path = os.path.join(cache_dir, f"v_{doc['video_name']}.mp4")
54
+ extensions = ["mp4", "webm", "mkv"]
55
+ for ext in extensions:
56
+ modified_path = video_path.replace("mp4", ext)
57
+ if os.path.exists(modified_path):
58
+ return [modified_path]
59
+ sys.exit(f"video path:{video_path} does not exist, please check")
60
+
61
+
62
+ # This is the place where format the question
63
+ def activitynetqa_doc_to_text(doc, lmms_eval_specific_kwargs=None):
64
+ if lmms_eval_specific_kwargs is None:
65
+ lmms_eval_specific_kwargs = {}
66
+ pre_prompt = ""
67
+ post_prompt = ""
68
+ if "pre_prompt" in lmms_eval_specific_kwargs:
69
+ pre_prompt = lmms_eval_specific_kwargs["pre_prompt"]
70
+ if "post_prompt" in lmms_eval_specific_kwargs:
71
+ post_prompt = lmms_eval_specific_kwargs["post_prompt"]
72
+
73
+ raw_question = doc["question"]
74
+ question = raw_question.capitalize() + "?"
75
+
76
+ # type_specific_prompts = {
77
+ # '3': "Please answer with 'yes' or 'no'.",
78
+ # '4': "Please state the color as a single word.",
79
+ # '7': "Please give the numerical answer."
80
+ # }
81
+
82
+ # doc_type = str(doc['type'])
83
+ # type_specific_prompt = type_specific_prompts.get(doc_type, "")
84
+
85
+ # return f"{pre_prompt}{question} {type_specific_prompt}{post_prompt}"
86
+ return f"{pre_prompt}{question}{post_prompt}"
87
+
88
+
89
+ def activitynetqa_doc_to_answer(doc):
90
+ return doc["answer"]
91
+
92
+
93
+ def get_eval(question, answer, pred, max_tokens: int, retries: int = 5):
94
+ global headers
95
+
96
+ messages = [
97
+ {
98
+ "role": "system",
99
+ "content": "You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. "
100
+ "Your task is to compare the predicted answer with the correct answer and determine if they match meaningfully. Here's how you can accomplish the task:"
101
+ "------"
102
+ "##INSTRUCTIONS: "
103
+ "- Focus on the meaningful match between the predicted answer and the correct answer.\n"
104
+ "- Consider synonyms or paraphrases as valid matches.\n"
105
+ "- Evaluate the correctness of the prediction compared to the answer.",
106
+ },
107
+ {
108
+ "role": "user",
109
+ "content": f"Please evaluate the following video-based question-answer pair:\n\n"
110
+ f"Question: {question}\n"
111
+ f"Correct Answer: {answer}\n"
112
+ f"Predicted Answer: {pred}\n\n"
113
+ "Provide your evaluation only as a yes/no and score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. "
114
+ "Please generate the response in the form of a Python dictionary string with keys 'pred' and 'score', where value of 'pred' is a string of 'yes' or 'no' and value of 'score' is in INTEGER, not STRING."
115
+ "DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. "
116
+ "For example, your response should look like this: {'pred': 'yes', 'score': 4.8}.",
117
+ },
118
+ ]
119
+
120
+ payload = {
121
+ "model": GPT_EVAL_MODEL_NAME,
122
+ "messages": messages,
123
+ "temperature": 0,
124
+ "max_tokens": max_tokens,
125
+ }
126
+
127
+ for attempt in range(retries):
128
+ try:
129
+ response = requests.post(API_URL, headers=headers, json=payload, timeout=60)
130
+ response.raise_for_status() # Raises HTTPError for bad responses
131
+ try:
132
+ response_data = response.json() # Attempt to parse JSON
133
+ except requests.exceptions.JSONDecodeError:
134
+ eval_logger.error(f"JSON decode error on attempt {attempt + 1}. Response text: {response.text}")
135
+ continue # Skip to next retry
136
+ content = response_data["choices"][0]["message"]["content"].strip()
137
+ if content != "":
138
+ return content, response_data["model"]
139
+ # Handle HTTP errors separately
140
+ except requests.exceptions.HTTPError as e:
141
+ eval_logger.error(f"HTTP error on attempt {attempt + 1}: {e}")
142
+ # Handle other requests-related errors
143
+ except requests.exceptions.RequestException as e:
144
+ eval_logger.error(f"Request exception on attempt {attempt + 1}: {e}")
145
+ except Exception as e:
146
+ eval_logger.error(f"Unexpected error on attempt {attempt + 1}: {e}")
147
+
148
+ # Handle other unexpected errors
149
+ if attempt < retries - 1:
150
+ time.sleep(NUM_SECONDS_TO_SLEEP)
151
+ else: # If this was the last attempt, log and return empty
152
+ eval_logger.error(f"All {retries} attempts failed. Last error message: {e}")
153
+ return "", ""
154
+
155
+ return "", ""
156
+
157
+
158
+ def parse_score(review):
159
+ try:
160
+ # Convert the string representation of a dictionary to an actual dictionary
161
+ review = "{" + review.split("{")[1].split("}")[0] + "}"
162
+ review_dict = ast.literal_eval(review)
163
+ # import pdb;pdb.set_trace()
164
+ score_match = review_dict["score"]
165
+ score = int(score_match)
166
+ pred = review_dict["pred"]
167
+ if "yes" in pred.lower():
168
+ pred = "yes"
169
+ elif "no" in pred.lower():
170
+ pred = "no"
171
+ # pred = review_dict.get("pred", "no")
172
+ # score = review_dict.get("score", 0)
173
+ return [pred, score]
174
+ except SyntaxError as e:
175
+ eval_logger.error(f"Syntax error parsing the review string: {e}. Review content: {review}")
176
+ except ValueError as e:
177
+ eval_logger.error(f"Value error parsing the review string: {e}. Review content: {review}")
178
+ except Exception as e:
179
+ eval_logger.error(f"Unexpected error parsing the review string: {e}. Review content: {review}")
180
+
181
+
182
+ def activitynetqa_process_results(doc, result):
183
+ """
184
+ Args:
185
+ doc: a instance of the eval dataset
186
+ results: [pred]
187
+ Returns:
188
+ a dictionary
189
+ """
190
+ try:
191
+ question = doc["question"]
192
+ answer = doc["answer"]
193
+ pred = result[0]
194
+
195
+ # Assume get_eval returns a review and the model name, and parse_score parses this review
196
+ review, model_name = get_eval(question, answer, pred, 64)
197
+ scores = parse_score(review)
198
+ except Exception as e:
199
+ eval_logger.error(f"Error for Question ID: {doc.get('question_id', 'Unknown')}: {e}")
200
+ review = "Failed to Get a Proper Review."
201
+ model_name = "Failed Request"
202
+ scores = ["no", 0]
203
+
204
+ return {
205
+ "gpt_eval_score": {"video_name": doc["video_name"], "question": doc["question"], "answer": doc["answer"], "pred": pred, "question_id": doc["question_id"], "type": doc["type"], "Correctness": scores[0], "score": scores[1]},
206
+ "gpt_eval_accuracy": {"video_name": doc["video_name"], "question": doc["question"], "answer": doc["answer"], "pred": pred, "question_id": doc["question_id"], "type": doc["type"], "Correctness": scores[0], "score": scores[1]},
207
+ }
208
+
209
+
210
+ def activitynetqa_gpt_eval(results, args):
211
+ """
212
+ Process the result file containing predictions, score them using GPT,
213
+ and save the results with added scores and correctness fields to a new file.
214
+
215
+ Args:
216
+ result_file_path: path to the JSON file with results to be evaluated
217
+ eval_file_path: path to save the JSON file with evaluated results
218
+ """
219
+
220
+ evaluated_results = []
221
+
222
+ # Process each result to generate scores
223
+ for data_dict in results:
224
+ try:
225
+ question = data_dict.get("Q", "")
226
+ answer = data_dict.get("A", "")
227
+ pred = data_dict.get("pred", "")
228
+
229
+ # Assume get_eval returns a review and the model name, and parse_score parses this review
230
+ review, model_name = get_eval(question, answer, pred, 64)
231
+ scores = parse_score(review)
232
+ except Exception as e:
233
+ eval_logger.error(f"Error for Question ID: {data_dict.get('question_id', 'Unknown')}: {e}")
234
+ review = "Failed to Get a Proper Review."
235
+ model_name = "Failed Request"
236
+ scores = ["no", 0]
237
+
238
+ # Update the dictionary with the new entries
239
+ updated_dict = {"video_name": data_dict["video_name"], "Correctness": scores[0], "score": scores[1], "Q": question, "A": answer, "pred": pred, "question_id": data_dict.get("question_id"), "type": data_dict.get("type")}
240
+ evaluated_results.append(updated_dict)
241
+
242
+ return evaluated_results
243
+
244
+
245
+ # Factory into different aggregate
246
+ def activitynetqa_aggregate_score(results, args):
247
+ yes_count = 0
248
+ no_count = 0
249
+ total_score = 0
250
+
251
+ # Iterate over the results to count correctness and sum scores
252
+ for result_dict in results:
253
+ if "yes" in result_dict["Correctness"].lower():
254
+ yes_count += 1
255
+ elif "no" in result_dict["Correctness"].lower():
256
+ no_count += 1
257
+
258
+ total_score += int(result_dict["score"])
259
+
260
+ # Calculate accuracy and average score
261
+ accuracy = yes_count / (yes_count + no_count) if (yes_count + no_count) > 0 else 0
262
+ average_score = total_score / len(results) if results else 0
263
+ eval_logger.info(f"Accuracy: {accuracy}")
264
+ eval_logger.info(f"Average Score: {average_score}")
265
+ return average_score
266
+
267
+
268
+ def activitynetqa_aggregate_accuracy(results, args):
269
+ yes_count = 0
270
+ no_count = 0
271
+ total_score = 0
272
+
273
+ # Iterate over the results to count correctness and sum scores
274
+ for result_dict in results:
275
+ if "yes" in result_dict["Correctness"].lower():
276
+ yes_count += 1
277
+ elif "no" in result_dict["Correctness"].lower():
278
+ no_count += 1
279
+
280
+ total_score += int(result_dict["score"])
281
+
282
+ # Calculate accuracy and average score
283
+ accuracy = yes_count / (yes_count + no_count) if (yes_count + no_count) > 0 else 0
284
+ average_score = total_score / len(results) if results else 0
285
+ eval_logger.info(f"Accuracy: {accuracy}")
286
+ eval_logger.info(f"Average Score: {average_score}")
287
+ return accuracy * 100
LaViDa/eval/lmms_eval/tasks/perceptiontest/test/perceptiontest_mcppl.yaml ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ dataset_name: "mc_question"
2
+ task: "perceptiontest_test_mcppl"
3
+ test_split: test
4
+ output_type: multiple_choice
5
+ doc_to_visual: !function utils.perceptiontest_doc_to_visual
6
+ doc_to_text: "question"
7
+ doc_to_target: !function utils.perceptiontest_doc_to_answer_mc
8
+ doc_to_choice: !function utils.perceptiontest_doc_to_choice
9
+ process_results: !function utils.perceptiontest_process_results_mc_ppl
10
+ metric_list:
11
+ - metric: submission
12
+ aggregation: !function utils.perceptiontest_aggregate_mc_ppl
13
+ higher_is_better: true
14
+ include: _default_template_yaml
LaViDa/eval/lmms_eval/tasks/video_detail_description/_default_template_yaml ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ dataset_path: lmms-lab/VideoDetailDescription
2
+ dataset_kwargs:
3
+ token: True
4
+ video: True
5
+ cache_dir: videochatgpt
6
+ lmms_eval_specific_kwargs:
7
+ default:
8
+ pre_prompt: ""
9
+ post_prompt: ""
10
+
11
+ metadata:
12
+ version: 0.0
13
+ gpt_eval_model_name: gpt-3.5-turbo-0613
LaViDa/eval/lmms_eval/tasks/video_detail_description/utils.py ADDED
@@ -0,0 +1,221 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import ast
2
+ import datetime
3
+ import json
4
+ import os
5
+ import sys
6
+ import time
7
+ from pathlib import Path
8
+
9
+ import requests
10
+ import yaml
11
+
12
+ import lmms_eval.tasks._task_utils.file_utils as file_utils
13
+
14
+ with open(Path(__file__).parent / "_default_template_yaml", "r") as f:
15
+ raw_data = f.readlines()
16
+ safe_data = []
17
+ for i, line in enumerate(raw_data):
18
+ # remove function definition since yaml load cannot handle it
19
+ if "!function" not in line:
20
+ safe_data.append(line)
21
+
22
+ config = yaml.safe_load("".join(safe_data))
23
+
24
+
25
+ NUM_SECONDS_TO_SLEEP = 5
26
+
27
+ GPT_EVAL_MODEL_NAME = config["metadata"]["gpt_eval_model_name"]
28
+
29
+ API_TYPE = os.getenv("API_TYPE", "openai")
30
+
31
+ if API_TYPE == "openai":
32
+ API_URL = os.getenv("OPENAI_API_URL", "https://api.openai.com/v1/chat/completions")
33
+ API_KEY = os.getenv("OPENAI_API_KEY", "YOUR_API_KEY")
34
+ headers = {
35
+ "Authorization": f"Bearer {API_KEY}",
36
+ "Content-Type": "application/json",
37
+ }
38
+
39
+ # A bit ugly here
40
+ # But the idea is that we will unzip all the zip files
41
+ # To HF HOME cache dir
42
+ # And load it here
43
+ HF_HOME = os.environ["HF_HOME"]
44
+ cache_dir = config["dataset_kwargs"]["cache_dir"]
45
+ cache_dir = os.path.join(HF_HOME, cache_dir)
46
+ cache_dir = os.path.join(cache_dir, "Test_Videos")
47
+
48
+ from loguru import logger as eval_logger
49
+
50
+
51
+ # Pass in video path here
52
+ # Can only work correctly with video llm
53
+ def video_detail_description_doc_to_visual(doc):
54
+ video_path = doc["video_name"] + ".mp4"
55
+ video_path = os.path.join(cache_dir, video_path)
56
+ if os.path.exists(video_path):
57
+ video_path = video_path
58
+ elif os.path.exists(video_path.replace("mp4", "MP4")):
59
+ video_path = video_path.replace("mp4", "MP4")
60
+ elif os.path.exists(video_path.replace("mp4", "mkv")):
61
+ video_path = video_path.replace("mp4", "mkv")
62
+ else:
63
+ sys.exit(f"video path:{video_path} does not exist, please check")
64
+ return [video_path]
65
+
66
+
67
+ # format the question
68
+ def video_detail_description_doc_to_text(doc, lmms_eval_specific_kwargs=None):
69
+ if lmms_eval_specific_kwargs is None:
70
+ lmms_eval_specific_kwargs = {}
71
+ pre_prompt = ""
72
+ post_prompt = ""
73
+ if "pre_prompt" in lmms_eval_specific_kwargs:
74
+ pre_prompt = lmms_eval_specific_kwargs["pre_prompt"]
75
+ if "post_prompt" in lmms_eval_specific_kwargs:
76
+ post_prompt = lmms_eval_specific_kwargs["post_prompt"]
77
+
78
+ question = doc["question"]
79
+ return f"{pre_prompt}{question}{post_prompt}"
80
+
81
+
82
+ def video_detail_description_doc_to_answer(doc):
83
+ return doc["answer"]
84
+
85
+
86
+ def get_eval_generic(question, answer, pred, max_tokens: int, retries: int = 5):
87
+ global headers
88
+
89
+ messages = [
90
+ {
91
+ "role": "system",
92
+ "content": "You are an intelligent chatbot designed for evaluating the detail orientation of generative outputs for video-based question-answer pairs. "
93
+ "Your task is to compare the predicted answer with the correct answer and determine its level of detail, considering both completeness and specificity. Here's how you can accomplish the task:"
94
+ "------"
95
+ "##INSTRUCTIONS: "
96
+ "- Check if the predicted answer covers all major points from the video. The response should not leave out any key aspects.\n"
97
+ "- Evaluate whether the predicted answer includes specific details rather than just generic points. It should provide comprehensive information that is tied to specific elements of the video.\n"
98
+ "- Consider synonyms or paraphrases as valid matches.\n"
99
+ "- Provide a single evaluation score that reflects the level of detail orientation of the prediction, considering both completeness and specificity.",
100
+ },
101
+ {
102
+ "role": "user",
103
+ "content": "Please evaluate the following video-based question-answer pair:\n\n"
104
+ f"Question: {question}\n"
105
+ f"Correct Answer: {answer}\n"
106
+ f"Predicted Answer: {pred}\n\n"
107
+ "Provide your evaluation only as a detail orientation score where the detail orientation score is an integer value between 0 and 5, with 5 indicating the highest level of detail orientation. "
108
+ "Please generate the response in the form of a Python dictionary string with keys 'score', where its value is the detail orientation score in INTEGER, not STRING."
109
+ "DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. "
110
+ "For example, your response should look like this: {'score': 4.8}.",
111
+ },
112
+ ]
113
+
114
+ payload = {
115
+ "model": GPT_EVAL_MODEL_NAME,
116
+ "messages": messages,
117
+ "temperature": 0,
118
+ "max_tokens": max_tokens,
119
+ # "response_format": {"type": "json_object"},
120
+ }
121
+
122
+ for attempt in range(retries):
123
+ try:
124
+ response = requests.post(API_URL, headers=headers, json=payload, timeout=60)
125
+ response.raise_for_status() # Raises HTTPError for bad responses
126
+ try:
127
+ response_data = response.json() # Attempt to parse JSON
128
+ except requests.exceptions.JSONDecodeError:
129
+ eval_logger.error(f"JSON decode error on attempt {attempt + 1}. Response text: {response.text}")
130
+ continue # Skip to next retry
131
+ content = response_data["choices"][0]["message"]["content"].strip()
132
+ if content != "":
133
+ return content, response_data["model"]
134
+ # Handle HTTP errors separately
135
+ except requests.exceptions.HTTPError as e:
136
+ eval_logger.error(f"HTTP error on attempt {attempt + 1}: {e}")
137
+ # Handle other requests-related errors
138
+ except requests.exceptions.RequestException as e:
139
+ eval_logger.error(f"Request exception on attempt {attempt + 1}: {e}")
140
+ except Exception as e:
141
+ eval_logger.error(f"Unexpected error on attempt {attempt + 1}: {e}")
142
+
143
+ if "Sorry! We've encountered an issue with repetitive patterns in your prompt. Please try again with a different prompt." in json.loads(response.content)["error"]["message"]:
144
+ eval_logger.error(f"Repetitive patterns in prompt. Drop this data.")
145
+ return "", ""
146
+
147
+ # Handle other unexpected errors
148
+ if attempt < retries - 1:
149
+ time.sleep(NUM_SECONDS_TO_SLEEP)
150
+ else: # If this was the last attempt, log and return empty
151
+ eval_logger.error(f"All {retries} attempts failed.")
152
+ return "", ""
153
+
154
+ return "", ""
155
+
156
+
157
+ def parse_score(review):
158
+ try:
159
+ # Convert the string representation of a dictionary to an actual dictionary
160
+ review_dict = ast.literal_eval(review)
161
+ score = review_dict.get("score", 0)
162
+ return int(score)
163
+ except SyntaxError as e:
164
+ eval_logger.error(f"Syntax error parsing the review string: {e}. Review content: {review}")
165
+ return 0
166
+ except ValueError as e:
167
+ eval_logger.error(f"Value error parsing the review string: {e}. Review content: {review}")
168
+ return 0
169
+ except Exception as e:
170
+ eval_logger.error(f"Unexpected error parsing the review string: {e}. Review content: {review}")
171
+ return 0
172
+
173
+
174
+ def gpt_eval(data_dict):
175
+ evaluated_results = []
176
+
177
+ try:
178
+ question = data_dict["question"]
179
+ answer = data_dict["answer"]
180
+ pred = data_dict["pred"]
181
+
182
+ # Assume get_eval returns a review and the model name, and parse_score parses this review
183
+ review, model_name = get_eval_generic(question, answer, pred, 64)
184
+ score = parse_score(review)
185
+ except Exception as e:
186
+ eval_logger.error(f"Error for Video Name: {data_dict.get('video_name', 'Unknown')}: {e}")
187
+ review = "Failed to Get a Proper Review."
188
+ model_name = ""
189
+ score = 0
190
+
191
+ # Update the dictionary with the new entries
192
+ updated_dict = {
193
+ "video_name": data_dict["video_name"],
194
+ "review": review,
195
+ "score": score,
196
+ }
197
+
198
+ return updated_dict
199
+
200
+
201
+ # Process result for evaluation in generic task
202
+ def video_detail_description_process_results_generic(doc, result):
203
+ pred = result[0]
204
+ doc["pred"] = pred
205
+ eval_results = gpt_eval(doc)
206
+
207
+ return {"gpt_eval_score": {"video_name": doc["video_name"], "question": doc["question"], "answer": doc["answer"], "pred": pred, "score": eval_results["score"], "review": eval_results["review"]}}
208
+
209
+
210
+ def video_detail_description_aggregate_score(results, args):
211
+ score = 0
212
+ for result in results:
213
+ eval_score = result["score"]
214
+ try:
215
+ eval_score = int(eval_score)
216
+ except:
217
+ eval_score = 0.0
218
+
219
+ score += eval_score
220
+
221
+ return score / len(results)
LaViDa/eval/lmms_eval/tasks/video_detail_description/video_detail_description.yaml ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ task: "video_dc499"
2
+ test_split: test
3
+ output_type: generate_until
4
+ doc_to_visual: !function utils.video_detail_description_doc_to_visual
5
+ doc_to_text: !function utils.video_detail_description_doc_to_text
6
+ doc_to_target: !function utils.video_detail_description_doc_to_answer
7
+ process_results: !function utils.video_detail_description_process_results_generic
8
+ metric_list:
9
+ - metric: gpt_eval_score
10
+ aggregation: !function utils.video_detail_description_aggregate_score
11
+ higher_is_better: true
12
+ include: _default_template_yaml
LaViDa/eval/lmms_eval/utils.py ADDED
@@ -0,0 +1,1018 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import collections
2
+ import datetime
3
+ import fnmatch
4
+ import functools
5
+ import hashlib
6
+ import importlib.util
7
+ import inspect
8
+ import json
9
+ import os
10
+ import pathlib
11
+ import re
12
+ import subprocess
13
+ import sys
14
+ import warnings
15
+ from typing import (
16
+ Any,
17
+ Callable,
18
+ Iterable,
19
+ Iterator,
20
+ List,
21
+ Literal,
22
+ Optional,
23
+ Tuple,
24
+ Type,
25
+ Union,
26
+ )
27
+
28
+ import yaml
29
+
30
+ warnings.simplefilter("ignore", category=DeprecationWarning)
31
+ warnings.filterwarnings("ignore")
32
+
33
+ import gc
34
+ from itertools import islice
35
+
36
+ import numpy as np
37
+ import pytz
38
+ import torch
39
+ import transformers
40
+ from jinja2 import BaseLoader, Environment, StrictUndefined
41
+ from loguru import logger as eval_logger
42
+
43
+ SPACING = " " * 47
44
+ HIGHER_IS_BETTER_SYMBOLS = {
45
+ True: "↑",
46
+ False: "↓",
47
+ }
48
+
49
+
50
+ def is_json(string):
51
+ try:
52
+ json.loads(string)
53
+ return True
54
+ except json.JSONDecodeError:
55
+ return False
56
+
57
+
58
+ def hash_string(string: str) -> str:
59
+ return hashlib.sha256(string.encode("utf-8")).hexdigest()
60
+
61
+
62
+ def escaped_split(text, sep_char, maxsplit=-1):
63
+ """Split text into a list on occurrences of the given separation
64
+ character `sep_char`. The separation character may be escaped by a
65
+ backslash to avoid splitting at that location.
66
+
67
+ The separation character must be a string of size 1.
68
+
69
+ If `maxsplit` is given, at most `maxsplit` splits are done (thus,
70
+ the list will have at most `maxsplit + 1` elements). If `maxsplit`
71
+ is not specified or less than 0, then there is no limit on the
72
+ number of splits (all possible splits are made).
73
+ """
74
+ assert len(sep_char) == 1, "separation string must be a single character for escaped splitting"
75
+
76
+ if maxsplit == 0:
77
+ return text
78
+ maxsplit = max(0, maxsplit)
79
+
80
+ return re.split(r"(?<!\\)" + sep_char, text, maxsplit)
81
+
82
+
83
+ def handle_arg_string(arg):
84
+ if arg.lower() == "true":
85
+ return True
86
+ elif arg.lower() == "false":
87
+ return False
88
+ elif arg.isnumeric():
89
+ return int(arg)
90
+ try:
91
+ return float(arg)
92
+ except ValueError:
93
+ return arg
94
+
95
+
96
+ def handle_non_serializable(o):
97
+ if isinstance(o, np.int64) or isinstance(o, np.int32):
98
+ return int(o)
99
+ elif isinstance(o, set):
100
+ return list(o)
101
+ else:
102
+ return str(o)
103
+
104
+
105
+ def sanitize_list(sub):
106
+ """
107
+ Takes possible nested list and recursively converts all inner component to strings
108
+ """
109
+ if isinstance(sub, list):
110
+ return [sanitize_list(item) for item in sub]
111
+ if isinstance(sub, tuple):
112
+ return tuple(sanitize_list(item) for item in sub)
113
+ else:
114
+ return str(sub)
115
+
116
+
117
+ def simple_parse_args_string(args_string):
118
+ """
119
+ Parses something like
120
+ args1=val1,arg2=val2
121
+ Into a dictionary
122
+ """
123
+ args_string = args_string.strip()
124
+ if not args_string:
125
+ return {}
126
+ arg_list = [arg for arg in args_string.split(",") if arg]
127
+ args_dict = {k: handle_arg_string(v) for k, v in [arg.split("=") for arg in arg_list]}
128
+ return args_dict
129
+
130
+
131
+ def join_iters(iters):
132
+ for iter in iters:
133
+ yield from iter
134
+
135
+
136
+ def chunks(iter, n: int = 0, fn=None):
137
+ """
138
+ Divides an iterable into chunks of specified size or based on a given function.
139
+ Useful for batching
140
+
141
+ Parameters:
142
+ - iter: The input iterable to be divided into chunks.
143
+ - n: An integer representing the size of each chunk. Default is 0.
144
+ - fn: A function that takes the current index and the iterable as arguments and returns the size of the chunk. Default is None.
145
+
146
+ Returns:
147
+ An iterator that yields chunks of the input iterable.
148
+
149
+ Example usage:
150
+ ```
151
+ data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
152
+ for chunk in chunks(data, 3):
153
+ print(chunk)
154
+ ```
155
+ Output:
156
+ ```
157
+ [1, 2, 3]
158
+ [4, 5, 6]
159
+ [7, 8, 9]
160
+ [10]
161
+ ```
162
+ """
163
+ arr = []
164
+ for i, x in enumerate(iter):
165
+ arr.append(x)
166
+ if len(arr) == (fn(i, iter) if fn else n):
167
+ yield arr
168
+ arr = []
169
+
170
+ if arr:
171
+ yield arr
172
+
173
+
174
+ def group(arr, fn):
175
+ res = collections.defaultdict(list)
176
+
177
+ for ob in arr:
178
+ res[fn(ob)].append(ob)
179
+
180
+ return list(res.values())
181
+
182
+
183
+ class MultiChoice:
184
+ def __init__(self, choices) -> None:
185
+ self.choices = choices
186
+
187
+ # Simple wildcard support (linux filename patterns)
188
+ def __contains__(self, values) -> bool:
189
+ for value in values.split(","):
190
+ if len(fnmatch.filter(self.choices, value)) == 0:
191
+ eval_logger.info(f"Available tasks to choose:")
192
+ for choice in self.choices:
193
+ eval_logger.info(f" - {choice}")
194
+ raise ValueError("'{}' is not in task list".format(value))
195
+ return True
196
+
197
+ def __iter__(self) -> Iterator:
198
+ for choice in self.choices:
199
+ yield choice
200
+
201
+
202
+ # Returns a list containing all values of the source_list that
203
+ # match at least one of the patterns
204
+ def pattern_match(patterns, source_list):
205
+ if type(patterns) == str:
206
+ patterns = [patterns]
207
+
208
+ task_names = set()
209
+ for pattern in patterns:
210
+ try:
211
+ for matching in fnmatch.filter(source_list, pattern):
212
+ task_names.add(matching)
213
+ except Exception as e:
214
+ eval_logger.error(f"Error matching pattern {pattern}: {e}")
215
+ return sorted(list(task_names))
216
+
217
+
218
+ def general_detokenize(string):
219
+ string = string.replace(" n't", "n't")
220
+ string = string.replace(" )", ")")
221
+ string = string.replace("( ", "(")
222
+ string = string.replace('" ', '"')
223
+ string = string.replace(' "', '"')
224
+ string = re.sub(r" (['.,])", r"\1", string)
225
+ return string
226
+
227
+
228
+ def get_file_task_name(filename: str) -> str:
229
+ """
230
+ Given the sample results filenames, extracts and returns the task name.
231
+ """
232
+ return filename[filename.find("_") + 1 : filename.rfind("_")]
233
+
234
+
235
+ def get_file_datetime(filename: str) -> str:
236
+ """
237
+ Given the results and sample results filenames, extracts and returns the datetime.
238
+ """
239
+ return filename[filename.rfind("_") + 1 :].replace(".jsonl", "")
240
+
241
+
242
+ def sanitize_model_name(model_name: str, full_path: bool = False) -> str:
243
+ """
244
+ Given the model name, returns a sanitized version of it.
245
+ """
246
+ if full_path:
247
+ return re.sub(r"[\"<>:/\|\\?\*\[\]]+", "__", model_name)
248
+ else:
249
+ parts = model_name.split("/")
250
+ last_two = "/".join(parts[-2:]) if len(parts) > 1 else parts[-1] # accommondate for models that are in Hugging Face Hub format like lmms-lab/llava-onevision-qwen2-0.5b
251
+ return re.sub(r"[\"<>:/\|\\?\*\[\]]+", "__", last_two)
252
+
253
+
254
+ def sanitize_task_name(task_name: str) -> str:
255
+ """
256
+ Given the task name, returns a sanitized version of it.
257
+ """
258
+ return re.sub(r"\W", "_", task_name)
259
+
260
+
261
+ def get_latest_filename(filenames: List[str]) -> str:
262
+ """
263
+ Given a list of filenames, returns the filename with the latest datetime.
264
+ """
265
+ return max(filenames, key=lambda f: get_file_datetime(f))
266
+
267
+
268
+ def get_results_filenames(filenames: List[str]) -> List[str]:
269
+ """
270
+ Extracts filenames that correspond to aggregated results.
271
+ """
272
+ return [f for f in filenames if "results" in f and ".json" in f]
273
+
274
+
275
+ def get_sample_results_filenames(filenames: List[str]) -> List[str]:
276
+ """
277
+ Extracts filenames that correspond to sample results.
278
+ """
279
+ return [f for f in filenames if "/samples_" in f and ".json" in f]
280
+
281
+
282
+ def get_rolling_token_windows(token_list, prefix_token, max_seq_len, context_len):
283
+ """
284
+ - context_len allows for a rolling window context, allowing each prediction window to potentially
285
+ condition on some context
286
+
287
+ :param token_list: list
288
+ List of tokens to be PREDICTED
289
+ :param max_seq_len: int
290
+ max_seq_len of model (or max_seq_len we want to use)
291
+ :param context_len: int
292
+ Amount of desired token context for prediction. Needs to be at least 1.
293
+ :param prefix_token: token
294
+ Dummy token like <eos> so the first token has something to condition on
295
+ :return: generator
296
+ Generator of tuples
297
+ (input_tokens, pred_tokens)
298
+ Note: Score only the last len(pred_tokens) logits of the LMM
299
+ """
300
+ assert 1 <= context_len <= max_seq_len
301
+ if not token_list:
302
+ return
303
+ # +1 offset, going from input->preds
304
+ pred_len = max_seq_len - context_len + 1
305
+ predicted = 0
306
+
307
+ # Special handling for first window: predict all tokens
308
+ first_seq_len = min(max_seq_len, len(token_list))
309
+ yield ([prefix_token] + token_list[: first_seq_len - 1], token_list[:first_seq_len])
310
+ predicted += first_seq_len
311
+
312
+ while predicted < len(token_list):
313
+ window_pred_len = min(len(token_list) - predicted, pred_len)
314
+ window_end = predicted + window_pred_len
315
+
316
+ yield (
317
+ token_list[window_end - max_seq_len - 1 : window_end - 1],
318
+ token_list[window_end - window_pred_len : window_end],
319
+ )
320
+ predicted += window_pred_len
321
+
322
+
323
+ def make_disjoint_window(pair):
324
+ """Takes output from get_rolling_token_windows and makes the context not overlap with the continuation"""
325
+ a, b = pair
326
+ return a[: len(a) - (len(b) - 1)], b
327
+
328
+
329
+ class EnhancedJSONEncoder(json.JSONEncoder):
330
+ """
331
+ Provides a proper json encoding for the loggers and trackers json dumps.
332
+ Notably manages the json encoding of dataclasses.
333
+ """
334
+
335
+ def default(self, o):
336
+ if is_dataclass(o):
337
+ return asdict(o)
338
+ return super().default(o)
339
+
340
+
341
+ class Reorderer:
342
+ def __init__(self, arr: List[Any], fn: Callable) -> None:
343
+ """Reorder an array according to some function
344
+
345
+ Args:
346
+ arr (List[Any]): The initial array
347
+ fn (Callable[[Any], Any]): A function to determine the priority of elements
348
+ """
349
+ self.size = len(arr)
350
+ arr = list(enumerate(arr))
351
+ arr = group(arr, lambda x: fn(x[1]))
352
+ # arr = [([y[0] for y in x], x[0][1]) for x in arr]
353
+ # TODO: overhaul reorderer. It currently grouped requests by content but we don't want this
354
+ arr = [([y[0]], x[0][1]) for x in arr for y in x]
355
+ arr.sort(key=lambda x: fn(x[1]))
356
+
357
+ self.arr = arr
358
+
359
+ def get_reordered(self):
360
+ """Gets the reordered array
361
+
362
+ Returns:
363
+ List[Any]: The reordered array
364
+ """
365
+ return [x[1] for x in self.arr]
366
+
367
+ def get_original(self, newarr):
368
+ """Restores the original order of a new array based on the old array's order
369
+
370
+ Args:
371
+ newarr (List[Any]): The array to be restored
372
+
373
+ Returns:
374
+ List[Any]: The array restored to the original order
375
+ """
376
+ res = [None] * self.size
377
+ cov = [False] * self.size
378
+
379
+ for (inds, _), v in zip(self.arr, newarr):
380
+ for ind in inds:
381
+ res[ind] = v
382
+ cov[ind] = True
383
+
384
+ assert all(cov)
385
+
386
+ return res
387
+
388
+
389
+ class Grouper:
390
+ """
391
+ takes an array `arr` and function `fn` and returns a dictionary
392
+ with keys fn(ob) for each ob in `arr` and with values `self.arr[key]` a list of all
393
+ objects in `arr` satisfying `key == fn(ob)`.
394
+ """
395
+
396
+ def __init__(self, arr, fn) -> None:
397
+ # self.orig_arr = arr
398
+ self.size = len(arr)
399
+ arr = list(enumerate(arr))
400
+
401
+ def group_return_dict(arr, fn):
402
+ res = collections.defaultdict(list)
403
+
404
+ for ob in arr:
405
+ res[fn(ob)].append(ob)
406
+ return res
407
+
408
+ arr = group_return_dict(arr, lambda x: fn(x[1]))
409
+
410
+ # self.arr has format Dict[Tuple[int, <entry from orig. arr>]]
411
+ self.arr = arr
412
+ self._grouped = None
413
+
414
+ def get_grouped(self):
415
+ # return the contents but not indices for our grouped dict.
416
+ if self._grouped:
417
+ return self._grouped
418
+ grouped = {}
419
+ for key in self.arr.keys():
420
+ # drop the index from each element of self.arr
421
+ grouped[key] = [y[1] for y in self.arr[key]]
422
+ self._grouped = grouped
423
+ return grouped
424
+
425
+ def get_original(self, grouped_dict):
426
+ # take in a grouped dictionary with e.g. results for each key listed
427
+ # in the same order as the instances in `self.arr`, and
428
+ # return the results in the same (single list) order as `self.orig_arr`.
429
+ res = [None] * self.size
430
+ cov = [False] * self.size
431
+ # orig = [None] * self.size
432
+
433
+ assert grouped_dict.keys() == self.arr.keys()
434
+
435
+ for key in grouped_dict.keys():
436
+ for (ind, _), v in zip(self.arr[key], grouped_dict[key]):
437
+ res[ind] = v
438
+ cov[ind] = True
439
+ # orig[ind] = _
440
+
441
+ assert all(cov)
442
+ # assert orig == self.orig_arr
443
+
444
+ return res
445
+
446
+
447
+ def make_table(result_dict, column: str = "results", sort_results: bool = False):
448
+ """Generate table of results."""
449
+ from pytablewriter import LatexTableWriter, MarkdownTableWriter
450
+
451
+ if column == "results":
452
+ column_name = "Tasks"
453
+ elif column == "groups":
454
+ column_name = "Groups"
455
+
456
+ all_headers = [
457
+ column_name,
458
+ "Version",
459
+ "Filter",
460
+ "n-shot",
461
+ "Metric",
462
+ "",
463
+ "Value",
464
+ "",
465
+ "Stderr",
466
+ ]
467
+
468
+ md_writer = MarkdownTableWriter()
469
+ latex_writer = LatexTableWriter()
470
+ md_writer.headers = all_headers
471
+ latex_writer.headers = all_headers
472
+
473
+ values = []
474
+
475
+ keys = result_dict[column].keys()
476
+ if sort_results:
477
+ # sort entries alphabetically by task or group name.
478
+ # NOTE: we default here to false, because order matters for multi-level table printing a la mmlu.
479
+ # sorting here would mess that up
480
+ keys = sorted(keys)
481
+ for k in keys:
482
+ dic = result_dict[column][k]
483
+ version = result_dict["versions"].get(k, " N/A")
484
+ n = str(result_dict.get("n-shot", " ").get(k, " "))
485
+ higher_is_better = result_dict.get("higher_is_better", {}).get(k, {})
486
+
487
+ if "alias" in dic:
488
+ k = dic.pop("alias")
489
+
490
+ metric_items = dic.items()
491
+ metric_items = sorted(metric_items)
492
+
493
+ for (mf), v in metric_items:
494
+ m, _, f = mf.partition(",")
495
+ if m.endswith("_stderr"):
496
+ continue
497
+
498
+ hib = HIGHER_IS_BETTER_SYMBOLS.get(higher_is_better.get(m), "")
499
+
500
+ v = "%.4f" % v if isinstance(v, float) else v
501
+ if v == "" or v is None:
502
+ v = "N/A"
503
+
504
+ if m + "_stderr" + "," + f in dic:
505
+ # if dic[m + "_stderr" + "," + f] != []:
506
+ se = dic[m + "_stderr" + "," + f]
507
+ se = " N/A" if se == "N/A" or se == [] else "%.4f" % se
508
+ if v != []:
509
+ values.append([k, version, f, n, m, hib, v, "±", se])
510
+ else:
511
+ values.append([k, version, f, n, m, hib, v, "", ""])
512
+ # k = ""
513
+ # version = ""
514
+ md_writer.value_matrix = values
515
+ latex_writer.value_matrix = values
516
+
517
+ # todo: make latex table look good
518
+ # print(latex_writer.dumps())
519
+
520
+ return md_writer.dumps()
521
+
522
+
523
+ def positional_deprecated(fn):
524
+ """
525
+ A decorator to nudge users into passing only keyword args (`kwargs`) to the
526
+ wrapped function, `fn`.
527
+ """
528
+
529
+ @functools.wraps(fn)
530
+ def _wrapper(*args, **kwargs):
531
+ if len(args) != 1 if inspect.ismethod(fn) else 0:
532
+ print(f"WARNING: using {fn.__name__} with positional arguments is " "deprecated and will be disallowed in a future version of " "lmms-evaluation-harness!")
533
+ return fn(*args, **kwargs)
534
+
535
+ return _wrapper
536
+
537
+
538
+ @positional_deprecated
539
+ def find_test_root(start_path: pathlib.Path) -> pathlib.Path:
540
+ """
541
+ Search upward in the directory tree to a maximum of three layers
542
+ to find and return the package root (containing the 'tests' folder)
543
+ """
544
+ cur_path = start_path.resolve()
545
+ max_layers = 3
546
+ for _ in range(max_layers):
547
+ if (cur_path / "tests" / "test_version_stable.py").exists():
548
+ return cur_path
549
+ else:
550
+ cur_path = cur_path.parent.resolve()
551
+ raise FileNotFoundError(f"Unable to find package root within {max_layers} upwards" + f"of {start_path}")
552
+
553
+
554
+ @positional_deprecated
555
+ def run_task_tests(task_list: List[str]):
556
+ """
557
+ Find the package root and run the tests for the given tasks
558
+ """
559
+ import pytest
560
+
561
+ package_root = find_test_root(start_path=pathlib.Path(__file__))
562
+ task_string = " or ".join(task_list)
563
+ args = [
564
+ f"{package_root}/tests/test_version_stable.py",
565
+ f"--rootdir={package_root}",
566
+ "-k",
567
+ f"{task_string}",
568
+ ]
569
+ sys.path.append(str(package_root))
570
+ pytest_return_val = pytest.main(args)
571
+ if pytest_return_val:
572
+ raise ValueError(f"Not all tests for the specified tasks ({task_list}) ran successfully! Error code: {pytest_return_val}")
573
+
574
+
575
+ def get_git_commit_hash():
576
+ """
577
+ Gets the git commit hash of your current repo (if it exists).
578
+ Source: https://github.com/EleutherAI/gpt-neox/blob/b608043be541602170bfcfb8ec9bf85e8a0799e0/megatron/neox_arguments/neox_args.py#L42
579
+ """
580
+ try:
581
+ git_hash = subprocess.check_output(["git", "describe", "--always"]).strip()
582
+ git_hash = git_hash.decode()
583
+ except subprocess.CalledProcessError or FileNotFoundError:
584
+ # FileNotFoundError occurs when git not installed on system
585
+ git_hash = None
586
+ return git_hash
587
+
588
+
589
+ def get_datetime_str(timezone="Asia/Singapore"):
590
+ """
591
+ Gets the current datetime in UTC+8 timezone as a string.
592
+ """
593
+ # Default: UTC+8 timezone
594
+ tz = pytz.timezone(timezone)
595
+ utc_now = datetime.datetime.now(datetime.timezone.utc)
596
+ local_time = utc_now.astimezone(tz)
597
+ return local_time.strftime("%Y%m%d_%H%M%S")
598
+ return local_time.strftime("%Y%m%d_%H%M%S")
599
+
600
+
601
+ def sanitize_long_string(s, max_length=40):
602
+ if len(s) > max_length:
603
+ return s[: max_length // 2] + "..." + s[-max_length // 2 :]
604
+ return s
605
+
606
+
607
+ def ignore_constructor(loader, node):
608
+ return node
609
+
610
+
611
+ def import_function(loader, node):
612
+ function_name = loader.construct_scalar(node)
613
+ yaml_path = os.path.dirname(loader.name)
614
+
615
+ *module_name, function_name = function_name.split(".")
616
+ if isinstance(module_name, list):
617
+ module_name = ".".join(module_name)
618
+ module_path = os.path.normpath(os.path.join(yaml_path, "{}.py".format(module_name)))
619
+
620
+ spec = importlib.util.spec_from_file_location(module_name, module_path)
621
+ module = importlib.util.module_from_spec(spec)
622
+ spec.loader.exec_module(module)
623
+
624
+ function = getattr(module, function_name)
625
+ return function
626
+
627
+
628
+ def load_yaml_config(yaml_path=None, yaml_config=None, yaml_dir=None, mode="full"):
629
+ if mode == "simple":
630
+ constructor_fn = ignore_constructor
631
+ elif mode == "full":
632
+ constructor_fn = import_function
633
+
634
+ # Add the import_function constructor to the YAML loader
635
+ yaml.add_constructor("!function", constructor_fn)
636
+ if yaml_config is None:
637
+ with open(yaml_path, "rb") as file:
638
+ yaml_config = yaml.full_load(file)
639
+
640
+ if yaml_dir is None:
641
+ yaml_dir = os.path.dirname(yaml_path)
642
+
643
+ assert yaml_dir is not None
644
+ assert yaml_config is not None
645
+
646
+ if "include" in yaml_config:
647
+ include_path = yaml_config["include"]
648
+ del yaml_config["include"]
649
+
650
+ if isinstance(include_path, str):
651
+ include_path = [include_path]
652
+
653
+ # Load from the last one first
654
+ include_path.reverse()
655
+ final_yaml_config = {}
656
+ for path in include_path:
657
+ # Assumes that path is a full path.
658
+ # If not found, assume the included yaml
659
+ # is in the same dir as the original yaml
660
+ if not os.path.isfile(path):
661
+ path = os.path.join(yaml_dir, path)
662
+
663
+ try:
664
+ included_yaml_config = load_yaml_config(yaml_path=path, mode=mode)
665
+ final_yaml_config.update(included_yaml_config)
666
+ except Exception as ex:
667
+ # If failed to load, ignore
668
+ raise ex
669
+
670
+ final_yaml_config.update(yaml_config)
671
+ return final_yaml_config
672
+ return yaml_config
673
+
674
+
675
+ def regex_replace(string, pattern, repl, count: int = 0):
676
+ """Implements the `re.sub` function as a custom Jinja filter."""
677
+ return re.sub(pattern, repl, string, count=count)
678
+
679
+
680
+ env = Environment(loader=BaseLoader, undefined=StrictUndefined)
681
+ env.filters["regex_replace"] = regex_replace
682
+
683
+
684
+ def apply_template(template: str, doc: dict) -> str:
685
+ rtemplate = env.from_string(template)
686
+ return rtemplate.render(**doc)
687
+
688
+
689
+ def create_iterator(raw_iterator, rank, world_size, limit=None):
690
+ """
691
+ Method for creating a (potentially) sliced and limited
692
+ iterator from a raw document iterator. Used for splitting data
693
+ among ranks in multigpu setting or only pulling a sample of documents
694
+ """
695
+ return islice(raw_iterator, rank, limit, world_size)
696
+
697
+
698
+ def pad_and_concat(
699
+ max_length: int,
700
+ tensors: List[torch.Tensor],
701
+ padding_side: Literal["right", "left"] = "right",
702
+ ):
703
+ """
704
+ Method for padding a list of tensors given the maximum tensor
705
+ length in the batch. Used for batching inputs and continuations in
706
+ seq2seq models.
707
+ """
708
+ assert padding_side == "left" or padding_side == "right", f"Unrecognized padding type: '{padding_side}' not 'left' or 'right'"
709
+
710
+ for i, tensor in enumerate(tensors):
711
+ if len(tensor.shape) == 2:
712
+ tensor = tensor.squeeze(0) # squeeze, in case passed [1, seq] size
713
+ tensor_len = tensor.shape[0]
714
+ if tensor_len < max_length:
715
+ if padding_side == "right":
716
+ # right-pad
717
+ tensors[i] = torch.cat(
718
+ [
719
+ tensor, # [seq]
720
+ torch.zeros(
721
+ max_length - tensor_len,
722
+ dtype=torch.long,
723
+ device=tensor.device,
724
+ ), # [padding_length - seq]
725
+ ],
726
+ dim=0,
727
+ ).unsqueeze(0)
728
+ else:
729
+ # left-pad
730
+ tensors[i] = torch.cat(
731
+ [
732
+ torch.zeros(
733
+ max_length - tensor_len,
734
+ dtype=torch.long,
735
+ device=tensor.device,
736
+ ), # [padding_length - seq]
737
+ tensor, # [seq]
738
+ ],
739
+ dim=0,
740
+ ).unsqueeze(0)
741
+ else:
742
+ tensors[i] = tensor.unsqueeze(0)
743
+
744
+ return torch.cat(tensors, dim=0)
745
+
746
+
747
+ def clear_torch_cache() -> None:
748
+ gc.collect()
749
+ torch.cuda.empty_cache()
750
+
751
+
752
+ def get_dtype(dtype: Union[str, torch.dtype]) -> torch.dtype:
753
+ """Converts `dtype` from `str` to torch.dtype when possible. Does not use an instantiated HF AutoConfig"""
754
+ if isinstance(dtype, str) and dtype != "auto":
755
+ # Convert `str` args torch dtype: `float16` -> `torch.float16`
756
+ _torch_dtype = getattr(torch, dtype)
757
+ else:
758
+ _torch_dtype = dtype
759
+ return _torch_dtype
760
+
761
+
762
+ # Multi-token stopping criteria
763
+ class MultiTokenEOSCriteria(transformers.StoppingCriteria):
764
+ """Criteria to stop on the specified multi-token sequence."""
765
+
766
+ def __init__(
767
+ self,
768
+ sequence: str,
769
+ tokenizer: transformers.PreTrainedTokenizer,
770
+ initial_decoder_input_length: int,
771
+ batch_size: int,
772
+ ) -> None:
773
+ self.initial_decoder_input_length = initial_decoder_input_length
774
+ self.done_tracker = [False] * batch_size
775
+ self.sequence = sequence
776
+ self.sequence_ids = tokenizer.encode(sequence, add_special_tokens=False)
777
+ # we look back for 2 more tokens than it takes to encode our stop sequence
778
+ # because tokenizers suck, and a model might generate `['\n', '\n']` but our `sequence` is `['\n\n']`
779
+ # and we don't want to mistakenly not stop a generation because our
780
+ # (string) stop sequence was output in a different tokenization
781
+
782
+ # NOTE: there is a minor danger that this will end up looking back 2 tokens into the past, into the inputs to the model,
783
+ # and stopping generation immediately as a result. With only 2 extra tokens of lookback, this risk is minimized
784
+ self.sequence_id_len = len(self.sequence_ids) + 2
785
+ self.tokenizer = tokenizer
786
+
787
+ def __call__(self, input_ids, scores, **kwargs) -> bool:
788
+ # For efficiency, we compare the last n tokens where n is the number of tokens in the stop_sequence
789
+ lookback_ids_batch = input_ids[:, self.initial_decoder_input_length :][:, -self.sequence_id_len :]
790
+
791
+ lookback_tokens_batch = self.tokenizer.batch_decode(lookback_ids_batch)
792
+ for i, done in enumerate(self.done_tracker):
793
+ if not done:
794
+ self.done_tracker[i] = self.sequence in lookback_tokens_batch[i]
795
+ return False not in self.done_tracker
796
+
797
+
798
+ def stop_sequences_criteria(
799
+ tokenizer: transformers.PreTrainedTokenizer,
800
+ stop_sequences: List[str],
801
+ initial_decoder_input_length: int,
802
+ batch_size: int,
803
+ ) -> transformers.StoppingCriteriaList:
804
+ return transformers.StoppingCriteriaList(
805
+ [
806
+ *[MultiTokenEOSCriteria(sequence, tokenizer, initial_decoder_input_length, batch_size) for sequence in stop_sequences],
807
+ ]
808
+ )
809
+
810
+
811
+ # from more_itertools
812
+ def divide(iterable, n) -> List[Iterator]:
813
+ """Divide the elements from *iterable* into *n* parts, maintaining
814
+ order.
815
+
816
+ >>> group_1, group_2 = divide(2, [1, 2, 3, 4, 5, 6])
817
+ >>> list(group_1)
818
+ [1, 2, 3]
819
+ >>> list(group_2)
820
+ [4, 5, 6]
821
+
822
+ If the length of *iterable* is not evenly divisible by *n*, then the
823
+ length of the returned iterables will not be identical:
824
+
825
+ >>> children = divide(3, [1, 2, 3, 4, 5, 6, 7])
826
+ >>> [list(c) for c in children]
827
+ [[1, 2, 3], [4, 5], [6, 7]]
828
+
829
+ If the length of the iterable is smaller than n, then the last returned
830
+ iterables will be empty:
831
+
832
+ >>> children = divide(5, [1, 2, 3])
833
+ >>> [list(c) for c in children]
834
+ [[1], [2], [3], [], []]
835
+
836
+ This function will exhaust the iterable before returning and may require
837
+ significant storage. If order is not important, see :func:`distribute`,
838
+ which does not first pull the iterable into memory.
839
+
840
+ """
841
+ if n < 1:
842
+ raise ValueError("n must be at least 1")
843
+
844
+ try:
845
+ iterable[:0]
846
+ except TypeError:
847
+ seq = tuple(iterable)
848
+ else:
849
+ seq = iterable
850
+
851
+ q, r = divmod(len(seq), n)
852
+
853
+ ret = []
854
+ stop = 0
855
+ for i in range(1, n + 1):
856
+ start = stop
857
+ stop += q + 1 if i <= r else q
858
+ ret.append(iter(seq[start:stop]))
859
+
860
+ return ret
861
+
862
+
863
+ class Collator:
864
+ """
865
+ A class for reordering and batching elements of an array.
866
+
867
+ This class allows for sorting an array based on a provided sorting function, grouping elements based on a grouping function, and generating batches from the sorted and grouped data.
868
+ """
869
+
870
+ def __init__(
871
+ self,
872
+ arr: List,
873
+ sort_fn: Callable,
874
+ group_fn: Callable = lambda x: x[1],
875
+ grouping: bool = False,
876
+ ) -> None:
877
+ self.grouping = grouping
878
+ self.fn = sort_fn
879
+ self.group_fn = lambda x: group_fn(x[1]) # first index are enumerated indices
880
+ self.reorder_indices: List = []
881
+ self.size = len(arr)
882
+ self.arr_with_indices: Iterable[Any] = tuple(enumerate(arr)) # [indices, (arr)]
883
+ if self.grouping is True:
884
+ self.group_by_index()
885
+
886
+ def group_by_index(self) -> None:
887
+ self.arr_with_indices = self.group(self.arr_with_indices, fn=self.group_fn, values=False)
888
+
889
+ def get_batched(self, n: int = 1, batch_fn: Optional[Callable] = None) -> Iterator:
890
+ """
891
+ Generates and yields batches from the reordered array.
892
+
893
+ Parameters:
894
+ - n (int): The size of each batch. Defaults to 1.
895
+ - batch_fn (Optional[Callable[[int, Iterable], int]]): A function to determine the size of each batch. Defaults to None.
896
+
897
+ Yields:
898
+ Iterator: An iterator over batches of reordered elements.
899
+ """
900
+ if self.grouping:
901
+ for (
902
+ key,
903
+ values,
904
+ ) in self.arr_with_indices.items(): # type: ignore
905
+ values = self._reorder(values)
906
+ batch = self.get_chunks(values, n=n, fn=batch_fn)
907
+ yield from batch
908
+ else:
909
+ values = self._reorder(self.arr_with_indices) # type: ignore
910
+ batch = self.get_chunks(values, n=n, fn=batch_fn)
911
+ yield from batch
912
+
913
+ def _reorder(self, arr: Union[List, Tuple[Tuple[int, Any], ...]]) -> List:
914
+ """
915
+ Reorders the elements in the array based on the sorting function.
916
+
917
+ Parameters:
918
+ - arr (Union[List, Tuple[Tuple[int, Any], ...]]): The array or iterable to be reordered.
919
+
920
+ Yields:
921
+ List: Yields reordered elements one by one.
922
+ """
923
+ arr = sorted(arr, key=lambda x: self.fn(x[1]))
924
+ self.reorder_indices.extend([x[0] for x in arr])
925
+ yield from [x[1] for x in arr]
926
+
927
+ def get_original(self, newarr: List) -> List:
928
+ """
929
+ Restores the original order of elements from the reordered list.
930
+
931
+ Parameters:
932
+ - newarr (List): The reordered array.
933
+
934
+ Returns:
935
+ List: The array with elements restored to their original order.
936
+ """
937
+ res = [None] * self.size
938
+ cov = [False] * self.size
939
+
940
+ for ind, v in zip(self.reorder_indices, newarr):
941
+ res[ind] = v
942
+ cov[ind] = True
943
+
944
+ assert all(cov)
945
+
946
+ return res
947
+
948
+ def __len__(self):
949
+ return self.size
950
+
951
+ @staticmethod
952
+ def group(arr: Iterable, fn: Callable, values: bool = False) -> Iterable:
953
+ """
954
+ Groups elements of an iterable based on a provided function.
955
+
956
+ Parameters:
957
+ - arr (Iterable): The iterable to be grouped.
958
+ - fn (Callable): The function to determine the grouping.
959
+ - values (bool): If True, returns the values of the group. Defaults to False.
960
+
961
+ Returns:
962
+ Iterable: An iterable of grouped elements.
963
+ """
964
+ res = collections.defaultdict(list)
965
+ for ob in arr:
966
+ try:
967
+ hashable_dict = tuple(
968
+ (
969
+ key,
970
+ tuple(value) if isinstance(value, collections.abc.Iterable) else value,
971
+ )
972
+ for key, value in sorted(fn(ob).items())
973
+ )
974
+ res[hashable_dict].append(ob)
975
+ except TypeError:
976
+ res[fn(ob)].append(ob)
977
+ if not values:
978
+ return res
979
+ return res.values()
980
+
981
+ @staticmethod
982
+ def get_chunks(_iter, n: int = 0, fn=None):
983
+ """
984
+ Divides an iterable into chunks of specified size or based on a given function.
985
+ Useful for batching
986
+
987
+ Parameters:
988
+ - iter: The input iterable to be divided into chunks.
989
+ - n: An integer representing the size of each chunk. Default is 0.
990
+ - fn: A function that takes the current index and the iterable as arguments and returns the size of the chunk. Default is None.
991
+
992
+ Returns:
993
+ An iterator that yields chunks of the input iterable.
994
+
995
+ Example usage:
996
+ ```
997
+ data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
998
+ for chunk in chunks(data, 3):
999
+ print(chunk)
1000
+ ```
1001
+ Output:
1002
+ ```
1003
+ [1, 2, 3]
1004
+ [4, 5, 6]
1005
+ [7, 8, 9]
1006
+ [10]
1007
+ ```
1008
+ """
1009
+ arr = []
1010
+ _iter = tuple(_iter)
1011
+ for i, x in enumerate(_iter):
1012
+ arr.append(x)
1013
+ if len(arr) == (fn(i, _iter) if fn else n):
1014
+ yield arr
1015
+ arr = []
1016
+
1017
+ if arr:
1018
+ yield arr
LaViDa/eval/pyproject.toml ADDED
@@ -0,0 +1,161 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [tool.black]
2
+ line-length = 240
3
+
4
+ [build-system]
5
+ requires = ["setuptools>=42", "wheel", "setuptools_scm[tomli]>=6.3"]
6
+ build-backend = "setuptools.build_meta"
7
+
8
+ [project]
9
+ name = "lmms_eval"
10
+ version = "0.3.0"
11
+ authors = [
12
+ { name = "LMMMs-Lab Evaluation Team", email = "lmms-lab@outlook.com" },
13
+ ]
14
+ description = "A framework for evaluating large multi-modality language models"
15
+ readme = "README.md"
16
+ classifiers = [
17
+ "Programming Language :: Python :: 3",
18
+ "License :: OSI Approved :: MIT License",
19
+ "Operating System :: OS Independent",
20
+ ]
21
+ requires-python = ">=3.8"
22
+ license = { text = "MIT" }
23
+ dependencies = [
24
+ "accelerate>=0.29.1",
25
+ "black==24.1.0",
26
+ "isort==5.13.2",
27
+ "datasets>=2.16.1",
28
+ "evaluate>=0.4.0",
29
+ "httpx==0.23.3",
30
+ "jsonlines",
31
+ "numexpr",
32
+ "peft>=0.2.0",
33
+ "pybind11>=2.6.2",
34
+ "pytablewriter",
35
+ "sacrebleu>=1.5.0",
36
+ "sqlitedict==2.1.0",
37
+ "timm",
38
+ "einops",
39
+ "ftfy",
40
+ "openai",
41
+ "opencv-python-headless",
42
+ "av",
43
+ "hf_transfer",
44
+ "nltk",
45
+ "sentencepiece",
46
+ "yt-dlp",
47
+ "pycocoevalcap",
48
+ "tqdm-multiprocess",
49
+ "transformers-stream-generator",
50
+ "zstandard",
51
+ "pillow",
52
+ "pyyaml",
53
+ "sympy",
54
+ "latex2sympy2",
55
+ "mpmath",
56
+ "Jinja2",
57
+ "openpyxl",
58
+ "loguru",
59
+ "hf_transfer",
60
+ "tenacity==8.3.0",
61
+ "wandb>=0.16.0",
62
+ "tiktoken",
63
+ "pre-commit",
64
+ "pydantic",
65
+ "packaging",
66
+ "decord",
67
+ "zss",
68
+ "protobuf",
69
+ "sentence_transformers",
70
+ "tokenizers"
71
+ ]
72
+
73
+ [project.optional-dependencies]
74
+ audio = [
75
+ "more-itertools",
76
+ "editdistance",
77
+ "zhconv",
78
+ "librosa",
79
+ "soundfile"
80
+ ]
81
+ metrics = [
82
+ "pywsd",
83
+ "spacy",
84
+ "anls",
85
+ "rouge",
86
+ "capture_metric",
87
+ "Levenshtein",
88
+ ]
89
+ gemini = [
90
+ "google-generativeai",
91
+ ]
92
+ reka = [
93
+ "httpx==0.23.3",
94
+ "reka-api",
95
+ ]
96
+ qwen = [
97
+ "decord",
98
+ "qwen_vl_utils",
99
+ ]
100
+ mmsearch = [
101
+ "playwright",
102
+ "requests",
103
+ "matplotlib",
104
+ "duckduckgo_search",
105
+ "langchain",
106
+ "langchain-community",
107
+ "beautifulsoup4",
108
+ "FlagEmbedding",
109
+ "rouge",
110
+ ]
111
+ all = [
112
+ "gemini",
113
+ "reka",
114
+ "metrics",
115
+ "qwen",
116
+ "mmsearch"
117
+ ]
118
+
119
+ [tool.setuptools.packages.find]
120
+ include = ["lmms_eval*"]
121
+ exclude = [
122
+ "assets*",
123
+ "benchmark*",
124
+ "docs",
125
+ "dist*",
126
+ "playground*",
127
+ "scripts*",
128
+ "tests*",
129
+ "checkpoints*",
130
+ "project_checkpoints*",
131
+ "debug_checkpoints*",
132
+ "mlx_configs*",
133
+ "wandb*",
134
+ "notebooks*",
135
+ "logs*",
136
+ ]
137
+
138
+ [tool.wheel]
139
+ exclude = [
140
+ "assets*",
141
+ "benchmark*",
142
+ "docs",
143
+ "dist*",
144
+ "playground*",
145
+ "scripts*",
146
+ "tests*",
147
+ "checkpoints*",
148
+ "project_checkpoints*",
149
+ "debug_checkpoints*",
150
+ "mlx_configs*",
151
+ "wandb*",
152
+ "notebooks*",
153
+ "logs*",
154
+ ]
155
+
156
+ [project.scripts]
157
+ lmms-eval = "lmms_eval.__main__:cli_evaluate"
158
+
159
+ [project.urls]
160
+ Homepage = "https://lmms-lab.github.io"
161
+ Repository = "https://github.com/EvolvingLMMs-Lab/lmms-eval"
LaViDa/eval/run_coco_dream.sh ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ LLADA_VISION_ENCODER="google/siglip-so400m-patch14-384"
2
+ set -x
3
+ export TASKS=${TASKS:-"mme"}
4
+ export CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7,8
5
+ export DEBUG_PRINT_IMAGE_RES=1
6
+ # max_new_tokens
7
+ accelerate launch --num_processes=8 --main_process_port=25511\
8
+ -m lmms_eval \
9
+ --model llava_dream \
10
+ --model_args pretrained=$1,conv_template=dream,model_name=llava_dream \
11
+ --tasks $TASKS \
12
+ --batch_size 1 \
13
+ --gen_kwargs alg=topk_margin,prefix_lm=True,step_ratio=0.5,schedule=shift,schedule__shift=0.33 \
14
+ --log_samples \
15
+ --log_samples_suffix llava_llada \
16
+ --output_path ./logs/ --verbosity=DEBUG \
17
+ ${@:2} \
LaViDa/scripts/zero2_fused_adamw.json ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "fp16": {
3
+ "enabled": "auto",
4
+ "loss_scale": 0,
5
+ "loss_scale_window": 1000,
6
+ "initial_scale_power": 16,
7
+ "hysteresis": 2,
8
+ "min_loss_scale": 1
9
+ },
10
+ "bf16": {
11
+ "enabled": "auto"
12
+ },
13
+ "optimizer": {
14
+ "type": "AdamW",
15
+ "params": {
16
+ "lr": "auto",
17
+ "betas": "auto",
18
+ "eps": "auto",
19
+ "weight_decay": "auto"
20
+ }
21
+ },
22
+ "zero_optimization": {
23
+ "stage": 2,
24
+ "offload_optimizer": {
25
+ "device": "none",
26
+ "pin_memory": true
27
+ },
28
+ "allgather_partitions": true,
29
+ "allgather_bucket_size": 2e8,
30
+ "overlap_comm": true,
31
+ "reduce_scatter": true,
32
+ "reduce_bucket_size": 2e8,
33
+ "contiguous_gradients": true
34
+ },
35
+ "gradient_accumulation_steps": "auto",
36
+ "gradient_clipping": "auto",
37
+ "steps_per_print": 100,
38
+ "train_batch_size": "auto",
39
+ "train_micro_batch_size_per_gpu": "auto",
40
+ "wall_clock_breakdown": false
41
+ }
rl_lladav_video_reversal_wo_t_reward/results/all_rewards.csv ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ step,acc_reward,t_grpo_reward,unbiased_reward,response_length
2
+ 1,0.5449,0.0000,0.1125,127.96
3
+ step,acc_reward,t_grpo_reward,unbiased_reward,response_length
4
+ 1,0.5449,0.0000,0.1125,127.96
5
+ 2,0.5664,0.0000,0.1125,128.00
6
+ 2,0.5664,0.0000,0.1125,128.00
7
+ 3,0.5000,0.0000,0.1031,128.00
8
+ 3,0.5000,0.0000,0.1031,128.00
9
+ 4,0.5410,0.0000,0.1500,127.98
10
+ 4,0.5410,0.0000,0.1500,127.98
11
+ 5,0.4922,0.0000,0.0937,127.99
12
+ 5,0.4922,0.0000,0.0937,127.99
13
+ 6,0.5215,0.0000,0.1219,127.99
14
+ 6,0.5215,0.0000,0.1219,127.99
15
+ 7,0.4805,0.0000,0.0937,128.00
16
+ 7,0.4805,0.0000,0.0937,128.00
17
+ 8,0.4297,0.0000,0.1219,128.00
18
+ 8,0.4297,0.0000,0.1219,128.00
19
+ 9,0.5234,0.0000,0.1219,128.00
20
+ 9,0.5234,0.0000,0.1219,128.00
21
+ 10,0.5098,0.0000,0.1500,128.00
22
+ 10,0.5098,0.0000,0.1500,128.00
23
+ 11,0.5000,0.0000,0.1031,127.99
24
+ 11,0.5000,0.0000,0.1031,127.99
25
+ 12,0.5859,0.0000,0.1031,127.99
26
+ 12,0.5859,0.0000,0.1031,127.99
27
+ 13,0.4883,0.0000,0.0844,127.99
28
+ 13,0.4883,0.0000,0.0844,127.99
29
+ 14,0.5430,0.0000,0.1125,127.99
30
+ 14,0.5430,0.0000,0.1125,127.99
31
+ 15,0.5391,0.0000,0.1594,128.00
32
+ 15,0.5391,0.0000,0.1594,128.00
33
+ 16,0.4961,0.0000,0.1219,127.99
34
+ 16,0.4961,0.0000,0.1219,127.99
35
+ 17,0.5332,0.0000,0.0844,128.00
36
+ 17,0.5332,0.0000,0.0844,128.00
37
+ 18,0.5723,0.0000,0.1406,128.00
38
+ 18,0.5723,0.0000,0.1406,128.00
39
+ 19,0.6250,0.0000,0.1781,128.00
40
+ 19,0.6250,0.0000,0.1781,128.00
41
+ 20,0.5312,0.0000,0.1031,127.98
42
+ 20,0.5312,0.0000,0.1031,127.98
43
+ 21,0.5703,0.0000,0.1219,128.00
44
+ 21,0.5703,0.0000,0.1219,128.00
45
+ 22,0.4238,0.0000,0.1500,127.99
46
+ 22,0.4238,0.0000,0.1500,127.99
47
+ 23,0.5859,0.0000,0.0937,128.00
48
+ 23,0.5859,0.0000,0.0937,128.00
49
+ 24,0.6133,0.0000,0.1219,127.99
50
+ 24,0.6133,0.0000,0.1219,127.99
51
+ 25,0.5371,0.0000,0.0844,128.00
52
+ 25,0.5371,0.0000,0.0844,128.00
53
+ 26,0.5059,0.0000,0.1406,127.99
54
+ 26,0.5059,0.0000,0.1406,127.99
55
+ 27,0.6055,0.0000,0.1219,128.00
56
+ 27,0.6055,0.0000,0.1219,128.00
57
+ 28,0.5703,0.0000,0.1219,128.00
58
+ 28,0.5703,0.0000,0.1219,128.00
59
+ 29,0.6035,0.0000,0.1312,127.99
60
+ 29,0.6035,0.0000,0.1312,127.99
61
+ 30,0.6270,0.0000,0.1594,127.99
62
+ 30,0.6270,0.0000,0.1594,127.99
rl_lladav_video_reversal_wo_t_reward/results/results-eval-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train.txt ADDED
File without changes
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step11.txt ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ train step: 11 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5000 avg length: 127.99
2
+ train step: 11 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5000 avg length: 127.99
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step23.txt ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ train step: 23 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5859 avg length: 128.00
2
+ train step: 23 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5859 avg length: 128.00
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step6.txt ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ train step: 6 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5215 avg length: 127.99
2
+ train step: 6 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5215 avg length: 127.99
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step7.txt ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ train step: 7 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.4805 avg length: 128.00
2
+ train step: 7 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.4805 avg length: 128.00
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step9.txt ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ train step: 9 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5234 avg length: 128.00
2
+ train step: 9 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5234 avg length: 128.00