Add files using upload-large-folder tool
Browse files- LaViDa/eval/.github/issue_template.md +8 -0
- LaViDa/eval/.github/pull_request_template.md +16 -0
- LaViDa/eval/.github/workflows/lint.yml +22 -0
- LaViDa/eval/.pre-commit-config.yaml +13 -0
- LaViDa/eval/lmms_eval/__init__.py +0 -0
- LaViDa/eval/lmms_eval/__main__.py +550 -0
- LaViDa/eval/lmms_eval/api/__init__.py +0 -0
- LaViDa/eval/lmms_eval/api/filter.py +54 -0
- LaViDa/eval/lmms_eval/api/group.py +104 -0
- LaViDa/eval/lmms_eval/api/instance.py +29 -0
- LaViDa/eval/lmms_eval/api/metrics.py +606 -0
- LaViDa/eval/lmms_eval/api/model.py +221 -0
- LaViDa/eval/lmms_eval/api/registry.py +185 -0
- LaViDa/eval/lmms_eval/api/samplers.py +96 -0
- LaViDa/eval/lmms_eval/api/task.py +1630 -0
- LaViDa/eval/lmms_eval/caching/__init__.py +0 -0
- LaViDa/eval/lmms_eval/caching/cache.py +66 -0
- LaViDa/eval/lmms_eval/eval_server.py +0 -0
- LaViDa/eval/lmms_eval/evaluator.py +654 -0
- LaViDa/eval/lmms_eval/evaluator_utils.py +488 -0
- LaViDa/eval/lmms_eval/filters/__init__.py +43 -0
- LaViDa/eval/lmms_eval/filters/decontamination.py +23 -0
- LaViDa/eval/lmms_eval/filters/extraction.py +282 -0
- LaViDa/eval/lmms_eval/filters/selection.py +59 -0
- LaViDa/eval/lmms_eval/filters/transformation.py +48 -0
- LaViDa/eval/lmms_eval/loggers/__init__.py +2 -0
- LaViDa/eval/lmms_eval/loggers/evaluation_tracker.py +452 -0
- LaViDa/eval/lmms_eval/loggers/utils.py +127 -0
- LaViDa/eval/lmms_eval/loggers/wandb_logger.py +315 -0
- LaViDa/eval/lmms_eval/logging_utils.py +368 -0
- LaViDa/eval/lmms_eval/models/egogpt.py +472 -0
- LaViDa/eval/lmms_eval/models/minicpm_v.py +224 -0
- LaViDa/eval/lmms_eval/models/moviechat.py +452 -0
- LaViDa/eval/lmms_eval/models/video_llava.py +214 -0
- LaViDa/eval/lmms_eval/tasks/activitynetqa/utils.py +287 -0
- LaViDa/eval/lmms_eval/tasks/perceptiontest/test/perceptiontest_mcppl.yaml +14 -0
- LaViDa/eval/lmms_eval/tasks/video_detail_description/_default_template_yaml +13 -0
- LaViDa/eval/lmms_eval/tasks/video_detail_description/utils.py +221 -0
- LaViDa/eval/lmms_eval/tasks/video_detail_description/video_detail_description.yaml +12 -0
- LaViDa/eval/lmms_eval/utils.py +1018 -0
- LaViDa/eval/pyproject.toml +161 -0
- LaViDa/eval/run_coco_dream.sh +17 -0
- LaViDa/scripts/zero2_fused_adamw.json +41 -0
- rl_lladav_video_reversal_wo_t_reward/results/all_rewards.csv +62 -0
- rl_lladav_video_reversal_wo_t_reward/results/results-eval-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train.txt +0 -0
- rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step11.txt +2 -0
- rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step23.txt +2 -0
- rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step6.txt +2 -0
- rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step7.txt +2 -0
- rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step9.txt +2 -0
LaViDa/eval/.github/issue_template.md
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Before you open an issue, please check if a similar issue already exists or has been closed before.
|
| 2 |
+
|
| 3 |
+
### When you open an issue, please be sure to include the following
|
| 4 |
+
|
| 5 |
+
- [ ] A descriptive title: [xxx] XXXX
|
| 6 |
+
- [ ] A detailed description
|
| 7 |
+
|
| 8 |
+
Thank you for your contributions!
|
LaViDa/eval/.github/pull_request_template.md
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Before you open a pull-request, please check if a similar issue already exists or has been closed before.
|
| 2 |
+
|
| 3 |
+
### When you open a pull-request, please be sure to include the following
|
| 4 |
+
|
| 5 |
+
- [ ] A descriptive title: [xxx] XXXX
|
| 6 |
+
- [ ] A detailed description
|
| 7 |
+
|
| 8 |
+
If you meet the lint warnings, you can use following scripts to reformat code.
|
| 9 |
+
|
| 10 |
+
```sh
|
| 11 |
+
pip install pre-commit
|
| 12 |
+
pre-commit install
|
| 13 |
+
pre-commit run --all-files
|
| 14 |
+
```
|
| 15 |
+
|
| 16 |
+
Thank you for your contributions!
|
LaViDa/eval/.github/workflows/lint.yml
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: Lint
|
| 2 |
+
|
| 3 |
+
on: [push, pull_request]
|
| 4 |
+
|
| 5 |
+
jobs:
|
| 6 |
+
lint:
|
| 7 |
+
runs-on: ubuntu-latest
|
| 8 |
+
steps:
|
| 9 |
+
- uses: actions/checkout@v3
|
| 10 |
+
with:
|
| 11 |
+
submodules: true
|
| 12 |
+
fetch-depth: 0
|
| 13 |
+
- name: Set up Python
|
| 14 |
+
uses: actions/setup-python@v4
|
| 15 |
+
with:
|
| 16 |
+
python-version: '3.9'
|
| 17 |
+
- name: Install pre-commit hook
|
| 18 |
+
run: |
|
| 19 |
+
python -m pip install pre-commit
|
| 20 |
+
pre-commit install
|
| 21 |
+
- name: Linting
|
| 22 |
+
run: pre-commit run --all-files
|
LaViDa/eval/.pre-commit-config.yaml
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
repos:
|
| 2 |
+
- repo: https://github.com/psf/black
|
| 3 |
+
rev: 23.12.1
|
| 4 |
+
hooks:
|
| 5 |
+
- id: black
|
| 6 |
+
language_version: python3
|
| 7 |
+
args: ["--line-length=240"]
|
| 8 |
+
- repo: https://github.com/PyCQA/isort
|
| 9 |
+
rev: 5.13.2
|
| 10 |
+
hooks:
|
| 11 |
+
- id: isort
|
| 12 |
+
language_version: python3
|
| 13 |
+
args: ["--profile", "black"]
|
LaViDa/eval/lmms_eval/__init__.py
ADDED
|
File without changes
|
LaViDa/eval/lmms_eval/__main__.py
ADDED
|
@@ -0,0 +1,550 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import argparse
|
| 2 |
+
import datetime
|
| 3 |
+
import importlib
|
| 4 |
+
import json
|
| 5 |
+
import os
|
| 6 |
+
import sys
|
| 7 |
+
import traceback
|
| 8 |
+
import warnings
|
| 9 |
+
from functools import partial
|
| 10 |
+
|
| 11 |
+
import numpy as np
|
| 12 |
+
import yaml
|
| 13 |
+
|
| 14 |
+
warnings.simplefilter("ignore", category=DeprecationWarning)
|
| 15 |
+
|
| 16 |
+
import hashlib
|
| 17 |
+
from pathlib import Path
|
| 18 |
+
from typing import Union
|
| 19 |
+
|
| 20 |
+
from accelerate import Accelerator
|
| 21 |
+
from accelerate.utils import InitProcessGroupKwargs
|
| 22 |
+
from loguru import logger as eval_logger
|
| 23 |
+
|
| 24 |
+
from lmms_eval import evaluator, utils
|
| 25 |
+
from lmms_eval.api.registry import ALL_TASKS
|
| 26 |
+
from lmms_eval.evaluator import request_caching_arg_to_dict
|
| 27 |
+
from lmms_eval.loggers import EvaluationTracker, WandbLogger
|
| 28 |
+
from lmms_eval.tasks import TaskManager
|
| 29 |
+
from lmms_eval.utils import (
|
| 30 |
+
handle_non_serializable,
|
| 31 |
+
make_table,
|
| 32 |
+
simple_parse_args_string,
|
| 33 |
+
)
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def _int_or_none_list_arg_type(min_len: int, max_len: int, defaults: str, value: str, split_char: str = ","):
|
| 37 |
+
def parse_value(item):
|
| 38 |
+
item = item.strip().lower()
|
| 39 |
+
if item == "none":
|
| 40 |
+
return None
|
| 41 |
+
try:
|
| 42 |
+
return int(item)
|
| 43 |
+
except ValueError:
|
| 44 |
+
raise argparse.ArgumentTypeError(f"{item} is not an integer or None")
|
| 45 |
+
|
| 46 |
+
items = [parse_value(v) for v in value.split(split_char)]
|
| 47 |
+
num_items = len(items)
|
| 48 |
+
|
| 49 |
+
if num_items == 1:
|
| 50 |
+
# Makes downstream handling the same for single and multiple values
|
| 51 |
+
items = items * max_len
|
| 52 |
+
elif num_items < min_len or num_items > max_len:
|
| 53 |
+
raise argparse.ArgumentTypeError(f"Argument requires {max_len} integers or None, separated by '{split_char}'")
|
| 54 |
+
elif num_items != max_len:
|
| 55 |
+
logging.warning(f"Argument requires {max_len} integers or None, separated by '{split_char}'. " "Missing values will be filled with defaults.")
|
| 56 |
+
default_items = [parse_value(v) for v in defaults.split(split_char)]
|
| 57 |
+
items.extend(default_items[num_items:]) # extend items list with missing defaults
|
| 58 |
+
|
| 59 |
+
return items
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def check_argument_types(parser: argparse.ArgumentParser):
|
| 63 |
+
"""
|
| 64 |
+
Check to make sure all CLI args are typed, raises error if not
|
| 65 |
+
"""
|
| 66 |
+
for action in parser._actions:
|
| 67 |
+
if action.dest != "help" and not action.const:
|
| 68 |
+
if action.type is None:
|
| 69 |
+
raise ValueError(f"Argument '{action.dest}' doesn't have a type specified.")
|
| 70 |
+
else:
|
| 71 |
+
continue
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def _handle_non_serializable(o):
|
| 75 |
+
if isinstance(o, np.int64) or isinstance(o, np.int32):
|
| 76 |
+
return int(o)
|
| 77 |
+
elif isinstance(o, set):
|
| 78 |
+
return list(o)
|
| 79 |
+
else:
|
| 80 |
+
return str(o)
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def parse_eval_args() -> argparse.Namespace:
|
| 84 |
+
parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter)
|
| 85 |
+
parser.add_argument("--config", default="", help="Path to a yaml file specifying all eval arguments, will ignore cli arguments if specified")
|
| 86 |
+
parser.add_argument("--model", default="hf", help="Name of model e.g. `hf`")
|
| 87 |
+
parser.add_argument(
|
| 88 |
+
"--tasks",
|
| 89 |
+
default=None,
|
| 90 |
+
help="To get full list of tasks, use the command lmms-eval --tasks list",
|
| 91 |
+
)
|
| 92 |
+
parser.add_argument(
|
| 93 |
+
"--model_args",
|
| 94 |
+
default="",
|
| 95 |
+
help="String arguments for model, e.g. `pretrained=EleutherAI/pythia-160m,dtype=float32`",
|
| 96 |
+
)
|
| 97 |
+
parser.add_argument(
|
| 98 |
+
"--num_fewshot",
|
| 99 |
+
type=int,
|
| 100 |
+
default=None,
|
| 101 |
+
help="Number of examples in few-shot context",
|
| 102 |
+
)
|
| 103 |
+
parser.add_argument(
|
| 104 |
+
"--batch_size",
|
| 105 |
+
"-b",
|
| 106 |
+
type=str,
|
| 107 |
+
default=1,
|
| 108 |
+
metavar="auto|auto:N|N",
|
| 109 |
+
help="Acceptable values are 'auto', 'auto:N' or N, where N is an integer. Default 1.",
|
| 110 |
+
)
|
| 111 |
+
parser.add_argument(
|
| 112 |
+
"--max_batch_size",
|
| 113 |
+
type=int,
|
| 114 |
+
default=None,
|
| 115 |
+
metavar="N",
|
| 116 |
+
help="Maximal batch size to try with --batch_size auto.",
|
| 117 |
+
)
|
| 118 |
+
parser.add_argument(
|
| 119 |
+
"--device",
|
| 120 |
+
type=str,
|
| 121 |
+
default=None,
|
| 122 |
+
help="Device to use (e.g. cuda, cuda:0, cpu)",
|
| 123 |
+
)
|
| 124 |
+
parser.add_argument(
|
| 125 |
+
"--output_path",
|
| 126 |
+
default=None,
|
| 127 |
+
type=str,
|
| 128 |
+
metavar="= [dir/file.jsonl] [DIR]",
|
| 129 |
+
help="The path to the output file where the result metrics will be saved. If the path is a directory and log_samples is true, the results will be saved in the directory. Else the parent directory will be used.",
|
| 130 |
+
)
|
| 131 |
+
parser.add_argument(
|
| 132 |
+
"--limit",
|
| 133 |
+
type=float,
|
| 134 |
+
default=None,
|
| 135 |
+
help="Limit the number of examples per task. " "If <1, limit is a percentage of the total number of examples.",
|
| 136 |
+
)
|
| 137 |
+
parser.add_argument(
|
| 138 |
+
"--use_cache",
|
| 139 |
+
"-c",
|
| 140 |
+
type=str,
|
| 141 |
+
default=None,
|
| 142 |
+
metavar="DIR",
|
| 143 |
+
help="A path to a sqlite db file for caching model responses. `None` if not caching.",
|
| 144 |
+
)
|
| 145 |
+
parser.add_argument(
|
| 146 |
+
"--cache_requests",
|
| 147 |
+
type=str,
|
| 148 |
+
default=None,
|
| 149 |
+
choices=["true", "refresh", "delete"],
|
| 150 |
+
help="Speed up evaluation by caching the building of dataset requests. `None` if not caching.",
|
| 151 |
+
)
|
| 152 |
+
parser.add_argument(
|
| 153 |
+
"--check_integrity",
|
| 154 |
+
action="store_true",
|
| 155 |
+
help="Whether to run the relevant part of the test suite for the tasks",
|
| 156 |
+
)
|
| 157 |
+
parser.add_argument(
|
| 158 |
+
"--write_out",
|
| 159 |
+
"-w",
|
| 160 |
+
action="store_true",
|
| 161 |
+
default=False,
|
| 162 |
+
help="Prints the prompt for the first few documents.",
|
| 163 |
+
)
|
| 164 |
+
parser.add_argument(
|
| 165 |
+
"--log_samples",
|
| 166 |
+
action="store_true",
|
| 167 |
+
default=False,
|
| 168 |
+
help="If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis",
|
| 169 |
+
)
|
| 170 |
+
parser.add_argument(
|
| 171 |
+
"--wandb_log_samples",
|
| 172 |
+
action="store_true",
|
| 173 |
+
default=False,
|
| 174 |
+
help="If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis to Weights and Biases",
|
| 175 |
+
)
|
| 176 |
+
parser.add_argument(
|
| 177 |
+
"--log_samples_suffix",
|
| 178 |
+
type=str,
|
| 179 |
+
default="model_outputs",
|
| 180 |
+
help="Specify a suffix for the log_samples file name.",
|
| 181 |
+
)
|
| 182 |
+
parser.add_argument(
|
| 183 |
+
"--system_instruction",
|
| 184 |
+
type=str,
|
| 185 |
+
default=None,
|
| 186 |
+
help="System instruction to be used in the prompt",
|
| 187 |
+
)
|
| 188 |
+
parser.add_argument(
|
| 189 |
+
"--apply_chat_template",
|
| 190 |
+
action="store_true",
|
| 191 |
+
default=False,
|
| 192 |
+
help="If True, applies the chat template to the prompt",
|
| 193 |
+
)
|
| 194 |
+
parser.add_argument(
|
| 195 |
+
"--fewshot_as_multiturn",
|
| 196 |
+
action="store_true",
|
| 197 |
+
default=False,
|
| 198 |
+
help="If True, uses the fewshot as a multi-turn conversation",
|
| 199 |
+
)
|
| 200 |
+
parser.add_argument(
|
| 201 |
+
"--show_config",
|
| 202 |
+
action="store_true",
|
| 203 |
+
default=False,
|
| 204 |
+
help="If True, shows the the full config of all tasks at the end of the evaluation.",
|
| 205 |
+
)
|
| 206 |
+
parser.add_argument(
|
| 207 |
+
"--include_path",
|
| 208 |
+
type=str,
|
| 209 |
+
default=None,
|
| 210 |
+
help="Additional path to include if there are external tasks to include.",
|
| 211 |
+
)
|
| 212 |
+
parser.add_argument(
|
| 213 |
+
"--gen_kwargs",
|
| 214 |
+
default="",
|
| 215 |
+
help=("String arguments for model generation on greedy_until tasks," " e.g. `temperature=0,top_k=0,top_p=0`"),
|
| 216 |
+
)
|
| 217 |
+
parser.add_argument(
|
| 218 |
+
"--verbosity",
|
| 219 |
+
type=str,
|
| 220 |
+
default="INFO",
|
| 221 |
+
help="Log error when tasks are not registered.",
|
| 222 |
+
)
|
| 223 |
+
parser.add_argument(
|
| 224 |
+
"--wandb_args",
|
| 225 |
+
default="",
|
| 226 |
+
help="Comma separated string arguments passed to wandb.init, e.g. `project=lmms-eval,job_type=eval",
|
| 227 |
+
)
|
| 228 |
+
parser.add_argument(
|
| 229 |
+
"--timezone",
|
| 230 |
+
default="Asia/Singapore",
|
| 231 |
+
help="Timezone for datetime string, e.g. Asia/Singapore, America/New_York, America/Los_Angeles. You can check the full list via `import pytz; print(pytz.common_timezones)`",
|
| 232 |
+
)
|
| 233 |
+
parser.add_argument(
|
| 234 |
+
"--hf_hub_log_args",
|
| 235 |
+
type=str,
|
| 236 |
+
default="",
|
| 237 |
+
help="Comma separated string arguments passed to Hugging Face Hub's log function, e.g. `hub_results_org=EleutherAI,hub_repo_name=lm-eval-results`",
|
| 238 |
+
)
|
| 239 |
+
parser.add_argument(
|
| 240 |
+
"--predict_only",
|
| 241 |
+
"-x",
|
| 242 |
+
action="store_true",
|
| 243 |
+
default=False,
|
| 244 |
+
help="Use with --log_samples. Only model outputs will be saved and metrics will not be evaluated.",
|
| 245 |
+
)
|
| 246 |
+
default_seed_string = "0,1234,1234,1234"
|
| 247 |
+
parser.add_argument(
|
| 248 |
+
"--seed",
|
| 249 |
+
type=partial(_int_or_none_list_arg_type, 3, 4, default_seed_string),
|
| 250 |
+
default=default_seed_string, # for backward compatibility
|
| 251 |
+
help=(
|
| 252 |
+
"Set seed for python's random, numpy, torch, and fewshot sampling.\n"
|
| 253 |
+
"Accepts a comma-separated list of 4 values for python's random, numpy, torch, and fewshot sampling seeds, "
|
| 254 |
+
"respectively, or a single integer to set the same seed for all four.\n"
|
| 255 |
+
f"The values are either an integer or 'None' to not set the seed. Default is `{default_seed_string}` "
|
| 256 |
+
"(for backward compatibility).\n"
|
| 257 |
+
"E.g. `--seed 0,None,8,52` sets `random.seed(0)`, `torch.manual_seed(8)`, and fewshot sampling seed to 52. "
|
| 258 |
+
"Here numpy's seed is not set since the second value is `None`.\n"
|
| 259 |
+
"E.g, `--seed 42` sets all four seeds to 42."
|
| 260 |
+
),
|
| 261 |
+
)
|
| 262 |
+
parser.add_argument(
|
| 263 |
+
"--trust_remote_code",
|
| 264 |
+
action="store_true",
|
| 265 |
+
help="Sets trust_remote_code to True to execute code to create HF Datasets from the Hub",
|
| 266 |
+
)
|
| 267 |
+
parser.add_argument("--process_with_media", action="store_true", help="Whether you will process you dataset with audio, image. By default set to False" "In case some benchmarks need to be processed with media, set this flag to True.")
|
| 268 |
+
args = parser.parse_args()
|
| 269 |
+
return args
|
| 270 |
+
|
| 271 |
+
|
| 272 |
+
def cli_evaluate(args: Union[argparse.Namespace, None] = None) -> None:
|
| 273 |
+
if not args:
|
| 274 |
+
args = parse_eval_args()
|
| 275 |
+
|
| 276 |
+
# Check if no arguments were passed after parsing
|
| 277 |
+
if len(sys.argv) == 1:
|
| 278 |
+
print("┌────────────────────────────���──────────────────────────────────────────────────┐")
|
| 279 |
+
print("│ Please provide arguments to evaluate the model. e.g. │")
|
| 280 |
+
print("│ `lmms-eval --model llava --model_path liuhaotian/llava-v1.6-7b --tasks okvqa` │")
|
| 281 |
+
print("│ Use `lmms-eval --help` for more information. │")
|
| 282 |
+
print("└───────────────────────────────────────────────────────────────────────────────┘")
|
| 283 |
+
sys.exit(1)
|
| 284 |
+
|
| 285 |
+
if args.wandb_args:
|
| 286 |
+
if "name" not in args.wandb_args:
|
| 287 |
+
name = f"{args.model}_{args.model_args}_{utils.get_datetime_str(timezone=args.timezone)}"
|
| 288 |
+
name = utils.sanitize_long_string(name)
|
| 289 |
+
args.wandb_args += f",name={name}"
|
| 290 |
+
wandb_logger = WandbLogger(**simple_parse_args_string(args.wandb_args))
|
| 291 |
+
|
| 292 |
+
# reset logger
|
| 293 |
+
eval_logger.remove()
|
| 294 |
+
eval_logger.add(sys.stdout, colorize=True, level=args.verbosity)
|
| 295 |
+
eval_logger.info(f"Verbosity set to {args.verbosity}")
|
| 296 |
+
os.environ["VERBOSITY"] = args.verbosity
|
| 297 |
+
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
| 298 |
+
|
| 299 |
+
args_list = []
|
| 300 |
+
results_list = []
|
| 301 |
+
if args.config:
|
| 302 |
+
if not os.path.exists(args.config):
|
| 303 |
+
raise ValueError(f"Config file does not exist: {args.config}")
|
| 304 |
+
|
| 305 |
+
with open(args.config, "r") as file:
|
| 306 |
+
config_args = yaml.safe_load(file)
|
| 307 |
+
config_args = [config_args] if type(config_args) != list else config_args
|
| 308 |
+
# multiple configs, create args list first
|
| 309 |
+
for config in config_args:
|
| 310 |
+
args_copy = argparse.Namespace(**vars(args))
|
| 311 |
+
for key, value in config.items():
|
| 312 |
+
setattr(args_copy, key, value)
|
| 313 |
+
args_list.append(args_copy)
|
| 314 |
+
else:
|
| 315 |
+
args_list.append(args)
|
| 316 |
+
|
| 317 |
+
# initialize Accelerator
|
| 318 |
+
kwargs_handler = InitProcessGroupKwargs(timeout=datetime.timedelta(seconds=60000))
|
| 319 |
+
accelerator = Accelerator(kwargs_handlers=[kwargs_handler])
|
| 320 |
+
if accelerator.is_main_process:
|
| 321 |
+
is_main_process = True
|
| 322 |
+
else:
|
| 323 |
+
is_main_process = False
|
| 324 |
+
|
| 325 |
+
for args in args_list:
|
| 326 |
+
try:
|
| 327 |
+
# if is_main_process and args.wandb_args: # thoughtfully we should only init wandb once, instead of multiple ranks to avoid network traffics and unwanted behaviors.
|
| 328 |
+
# wandb_logger = WandbLogger()
|
| 329 |
+
|
| 330 |
+
results, samples = cli_evaluate_single(args)
|
| 331 |
+
results_list.append(results)
|
| 332 |
+
|
| 333 |
+
accelerator.wait_for_everyone()
|
| 334 |
+
if is_main_process and args.wandb_args:
|
| 335 |
+
try:
|
| 336 |
+
wandb_logger.post_init(results)
|
| 337 |
+
wandb_logger.log_eval_result()
|
| 338 |
+
if args.wandb_log_samples and samples is not None:
|
| 339 |
+
wandb_logger.log_eval_samples(samples)
|
| 340 |
+
except Exception as e:
|
| 341 |
+
eval_logger.info(f"Logging to Weights and Biases failed due to {e}")
|
| 342 |
+
# wandb_logger.finish()
|
| 343 |
+
|
| 344 |
+
except Exception as e:
|
| 345 |
+
if args.verbosity == "DEBUG":
|
| 346 |
+
raise e
|
| 347 |
+
else:
|
| 348 |
+
traceback.print_exc()
|
| 349 |
+
eval_logger.error(f"Error during evaluation: {e}. Please set `--verbosity=DEBUG` to get more information.")
|
| 350 |
+
results_list.append(None)
|
| 351 |
+
|
| 352 |
+
for args, results in zip(args_list, results_list):
|
| 353 |
+
# cli_evaluate will return none if the process is not the main process (rank 0)
|
| 354 |
+
if results is not None:
|
| 355 |
+
print(f"{args.model} ({args.model_args}), gen_kwargs: ({args.gen_kwargs}), limit: {args.limit}, num_fewshot: {args.num_fewshot}, " f"batch_size: {args.batch_size}")
|
| 356 |
+
print(make_table(results))
|
| 357 |
+
if "groups" in results:
|
| 358 |
+
print(make_table(results, "groups"))
|
| 359 |
+
|
| 360 |
+
if args.wandb_args:
|
| 361 |
+
wandb_logger.run.finish()
|
| 362 |
+
|
| 363 |
+
|
| 364 |
+
def cli_evaluate_single(args: Union[argparse.Namespace, None] = None) -> None:
|
| 365 |
+
selected_task_list = args.tasks.split(",") if args.tasks else None
|
| 366 |
+
|
| 367 |
+
if args.include_path is not None:
|
| 368 |
+
eval_logger.info(f"Including path: {args.include_path}")
|
| 369 |
+
task_manager = TaskManager(args.verbosity, include_path=args.include_path, model_name=args.model)
|
| 370 |
+
|
| 371 |
+
# update the evaluation tracker args with the output path and the HF token
|
| 372 |
+
if args.output_path:
|
| 373 |
+
args.hf_hub_log_args += f",output_path={args.output_path}"
|
| 374 |
+
if os.environ.get("HF_TOKEN", None):
|
| 375 |
+
args.hf_hub_log_args += f",token={os.environ.get('HF_TOKEN')}"
|
| 376 |
+
|
| 377 |
+
evaluation_tracker_args = simple_parse_args_string(args.hf_hub_log_args)
|
| 378 |
+
eval_logger.info(f"Evaluation tracker args: {evaluation_tracker_args}")
|
| 379 |
+
|
| 380 |
+
evaluation_tracker = EvaluationTracker(**evaluation_tracker_args)
|
| 381 |
+
|
| 382 |
+
if args.predict_only:
|
| 383 |
+
args.log_samples = True
|
| 384 |
+
if (args.log_samples or args.predict_only) and not args.output_path:
|
| 385 |
+
raise ValueError("Specify --output_path if providing --log_samples or --predict_only")
|
| 386 |
+
|
| 387 |
+
if args.fewshot_as_multiturn and args.apply_chat_template is False:
|
| 388 |
+
raise ValueError("If fewshot_as_multiturn is set, apply_chat_template must be set to True.")
|
| 389 |
+
|
| 390 |
+
if (args.num_fewshot is None or args.num_fewshot == 0) and args.fewshot_as_multiturn:
|
| 391 |
+
raise ValueError("If fewshot_as_multiturn is set, num_fewshot must be greater than 0.")
|
| 392 |
+
|
| 393 |
+
if args.include_path is not None:
|
| 394 |
+
eval_logger.info(f"Including path: {args.include_path}")
|
| 395 |
+
|
| 396 |
+
if "push_samples_to_hub" in evaluation_tracker_args and not args.log_samples:
|
| 397 |
+
eval_logger.warning("Pushing samples to the Hub requires --log_samples to be set. Samples will not be pushed to the Hub.")
|
| 398 |
+
|
| 399 |
+
if args.limit:
|
| 400 |
+
eval_logger.warning(" --limit SHOULD ONLY BE USED FOR TESTING." "REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.")
|
| 401 |
+
|
| 402 |
+
if os.environ.get("LMMS_EVAL_PLUGINS", None):
|
| 403 |
+
args.include_path = [args.include_path] if args.include_path else []
|
| 404 |
+
for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
|
| 405 |
+
package_tasks_location = importlib.util.find_spec(f"{plugin}.tasks").submodule_search_locations[0]
|
| 406 |
+
args.include_path.append(package_tasks_location)
|
| 407 |
+
|
| 408 |
+
if args.tasks is None:
|
| 409 |
+
eval_logger.error("Need to specify task to evaluate.")
|
| 410 |
+
sys.exit()
|
| 411 |
+
elif args.tasks == "list":
|
| 412 |
+
eval_logger.info("Available Tasks:\n - {}".format(f"\n - ".join(sorted(task_manager.list_all_tasks()))))
|
| 413 |
+
sys.exit()
|
| 414 |
+
elif args.tasks == "list_groups":
|
| 415 |
+
eval_logger.info(task_manager.list_all_tasks(list_subtasks=False, list_tags=False))
|
| 416 |
+
sys.exit()
|
| 417 |
+
elif args.tasks == "list_tags":
|
| 418 |
+
eval_logger.info(task_manager.list_all_tasks(list_groups=False, list_subtasks=False))
|
| 419 |
+
sys.exit()
|
| 420 |
+
elif args.tasks == "list_subtasks":
|
| 421 |
+
eval_logger.info(task_manager.list_all_tasks(list_groups=False, list_tags=False))
|
| 422 |
+
sys.exit()
|
| 423 |
+
elif args.tasks == "list_with_num":
|
| 424 |
+
log_message = (
|
| 425 |
+
"\n" + "=" * 70 + "\n" + "\n\tYou are trying to check all the numbers in each task." + "\n\tThis action will download the complete dataset." + "\n\tIf the results are not clear initially, call this again." + "\n\n" + "=" * 70
|
| 426 |
+
)
|
| 427 |
+
eval_logger.info(log_message)
|
| 428 |
+
for task_name in sorted(task_manager.list_all_tasks()):
|
| 429 |
+
try:
|
| 430 |
+
task_dict = get_task_dict([task_name], model_name="llava")
|
| 431 |
+
task_obj = task_dict[task_name]
|
| 432 |
+
if type(task_obj) == tuple:
|
| 433 |
+
group, task_obj = task_obj
|
| 434 |
+
if task_obj is None:
|
| 435 |
+
continue
|
| 436 |
+
eval_logger.info(f"\nTask : {task_obj.config.task}\n - #num : {len(task_obj.test_docs()) if task_obj.has_test_docs() else len(task_obj.validation_docs())}")
|
| 437 |
+
except Exception as e:
|
| 438 |
+
eval_logger.debug(f"\nTask : {task_name} fail to load \n Exception : \n {e}")
|
| 439 |
+
sys.exit()
|
| 440 |
+
else:
|
| 441 |
+
if os.path.isdir(args.tasks):
|
| 442 |
+
import glob
|
| 443 |
+
|
| 444 |
+
task_names = []
|
| 445 |
+
yaml_path = os.path.join(args.tasks, "*.yaml")
|
| 446 |
+
for yaml_file in glob.glob(yaml_path):
|
| 447 |
+
config = utils.load_yaml_config(yaml_file)
|
| 448 |
+
task_names.append(config)
|
| 449 |
+
else:
|
| 450 |
+
task_list = args.tasks.split(",")
|
| 451 |
+
task_names = task_manager.match_tasks(task_list)
|
| 452 |
+
for task in [task for task in task_list if task not in task_names]:
|
| 453 |
+
if os.path.isfile(task):
|
| 454 |
+
config = utils.load_yaml_config(task)
|
| 455 |
+
task_names.append(config)
|
| 456 |
+
task_missing = [task for task in task_list if task not in task_names and "*" not in task] # we don't want errors if a wildcard ("*") task name was used
|
| 457 |
+
|
| 458 |
+
if task_missing:
|
| 459 |
+
missing = ", ".join(task_missing)
|
| 460 |
+
eval_logger.error(
|
| 461 |
+
f"Tasks were not found: {missing}\n" f"{utils.SPACING}Try `lmms-eval --tasks list` for list of available tasks",
|
| 462 |
+
)
|
| 463 |
+
raise ValueError(
|
| 464 |
+
f"Tasks not found: {missing}. Try `lmms-eval --tasks {{list_groups,list_subtasks,list_tags,list}}` to list out all available names for task groupings; only (sub)tasks; tags; or all of the above, or pass '--verbosity DEBUG' to troubleshoot task registration issues."
|
| 465 |
+
)
|
| 466 |
+
|
| 467 |
+
eval_logger.info(f"Selected Tasks: {task_names}")
|
| 468 |
+
request_caching_args = request_caching_arg_to_dict(cache_requests=args.cache_requests)
|
| 469 |
+
datetime_str = utils.get_datetime_str(timezone=args.timezone)
|
| 470 |
+
|
| 471 |
+
results = evaluator.simple_evaluate(
|
| 472 |
+
model=args.model,
|
| 473 |
+
model_args=args.model_args,
|
| 474 |
+
tasks=task_names,
|
| 475 |
+
num_fewshot=args.num_fewshot,
|
| 476 |
+
batch_size=args.batch_size,
|
| 477 |
+
max_batch_size=args.max_batch_size,
|
| 478 |
+
device=args.device,
|
| 479 |
+
use_cache=args.use_cache,
|
| 480 |
+
limit=args.limit,
|
| 481 |
+
check_integrity=args.check_integrity,
|
| 482 |
+
write_out=args.write_out,
|
| 483 |
+
log_samples=args.log_samples,
|
| 484 |
+
evaluation_tracker=evaluation_tracker,
|
| 485 |
+
system_instruction=args.system_instruction,
|
| 486 |
+
apply_chat_template=args.apply_chat_template,
|
| 487 |
+
fewshot_as_multiturn=args.fewshot_as_multiturn,
|
| 488 |
+
gen_kwargs=args.gen_kwargs,
|
| 489 |
+
task_manager=task_manager,
|
| 490 |
+
verbosity=args.verbosity,
|
| 491 |
+
predict_only=args.predict_only,
|
| 492 |
+
random_seed=args.seed[0],
|
| 493 |
+
numpy_random_seed=args.seed[1],
|
| 494 |
+
torch_random_seed=args.seed[2],
|
| 495 |
+
fewshot_random_seed=args.seed[3],
|
| 496 |
+
cli_args=args,
|
| 497 |
+
datetime_str=datetime_str,
|
| 498 |
+
**request_caching_args,
|
| 499 |
+
)
|
| 500 |
+
|
| 501 |
+
if results is not None:
|
| 502 |
+
# breakpoint()
|
| 503 |
+
all_metrics = {}
|
| 504 |
+
for task,task_result in results['results'].items():
|
| 505 |
+
metrics ={}
|
| 506 |
+
for metric_key,metric_value in task_result.items():
|
| 507 |
+
if metric_key in ['alias']:
|
| 508 |
+
continue
|
| 509 |
+
if isinstance(metric_value, (int, float)):
|
| 510 |
+
metrics[metric_key] = metric_value
|
| 511 |
+
for k,v in metrics.items():
|
| 512 |
+
all_metrics[f'{task}/{k}'] = v
|
| 513 |
+
# breakpoint()
|
| 514 |
+
if os.environ.get('CONTROL_LOG_DIR'):
|
| 515 |
+
control_log_dir = os.environ.get('CONTROL_LOG_DIR')
|
| 516 |
+
with open(control_log_dir,'w') as f:
|
| 517 |
+
f.write( json.dumps(all_metrics))
|
| 518 |
+
|
| 519 |
+
if args.log_samples:
|
| 520 |
+
samples = results.pop("samples")
|
| 521 |
+
else:
|
| 522 |
+
samples = None
|
| 523 |
+
dumped = json.dumps(results, indent=4, default=_handle_non_serializable)
|
| 524 |
+
if args.show_config:
|
| 525 |
+
print(dumped)
|
| 526 |
+
|
| 527 |
+
batch_sizes = ",".join(map(str, results["config"]["batch_sizes"]))
|
| 528 |
+
|
| 529 |
+
evaluation_tracker.save_results_aggregated(results=results, samples=samples if args.log_samples else None, datetime_str=datetime_str)
|
| 530 |
+
|
| 531 |
+
if args.log_samples:
|
| 532 |
+
for task_name, config in results["configs"].items():
|
| 533 |
+
evaluation_tracker.save_results_samples(task_name=task_name, samples=samples[task_name])
|
| 534 |
+
|
| 535 |
+
if evaluation_tracker.push_results_to_hub or evaluation_tracker.push_samples_to_hub:
|
| 536 |
+
evaluation_tracker.recreate_metadata_card()
|
| 537 |
+
|
| 538 |
+
return results, samples
|
| 539 |
+
return None, None
|
| 540 |
+
|
| 541 |
+
|
| 542 |
+
def print_results(args, results):
|
| 543 |
+
print(f"{args.model} ({args.model_args}),\ngen_kwargs: ({args.gen_kwargs}),\nlimit: {args.limit},\nnum_fewshot: {args.num_fewshot},\nbatch_size: {args.batch_size}")
|
| 544 |
+
print(evaluator.make_table(results))
|
| 545 |
+
if "groups" in results:
|
| 546 |
+
print(evaluator.make_table(results, "groups"))
|
| 547 |
+
|
| 548 |
+
|
| 549 |
+
if __name__ == "__main__":
|
| 550 |
+
cli_evaluate()
|
LaViDa/eval/lmms_eval/api/__init__.py
ADDED
|
File without changes
|
LaViDa/eval/lmms_eval/api/filter.py
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from dataclasses import dataclass
|
| 2 |
+
from typing import List
|
| 3 |
+
|
| 4 |
+
from datasets import Dataset
|
| 5 |
+
|
| 6 |
+
from lmms_eval.api.instance import Instance
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
class Filter:
|
| 10 |
+
"""
|
| 11 |
+
Filter classes operate on a per-task level.
|
| 12 |
+
They take all model outputs (`instance.resps` for all `task.instances`)
|
| 13 |
+
across all instances of a task, and perform operations.
|
| 14 |
+
In a single run, one can configure any number of separate filters or lists of filters.
|
| 15 |
+
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
def __init__(self, *args, **kwargs) -> None:
|
| 19 |
+
"""
|
| 20 |
+
Can define custom behavior here, if an individual instantiation of a Filter class should have state.
|
| 21 |
+
"""
|
| 22 |
+
|
| 23 |
+
def apply(self, resps, docs):
|
| 24 |
+
"""
|
| 25 |
+
Defines the operation to perform on a list of the `inst.resps` properties of `Instance` objects.
|
| 26 |
+
Should return the list of (filtered) response lists *in the same order as they were input*, e.g.
|
| 27 |
+
if pass in [<inst.resps for instance 0>, <inst.resps for instance 1>] should return
|
| 28 |
+
[<filtered resps for instance 0>, <filtered resps for instance 1>]
|
| 29 |
+
"""
|
| 30 |
+
return resps
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
@dataclass
|
| 34 |
+
class FilterEnsemble:
|
| 35 |
+
"""
|
| 36 |
+
FilterEnsemble creates a pipeline applying multiple filters.
|
| 37 |
+
Its intended usage is to stack multiple post-processing steps in order.
|
| 38 |
+
`task.apply_filters` should use a list of FilterEnsemble classes that it stores, to apply each
|
| 39 |
+
pipeline separately.
|
| 40 |
+
"""
|
| 41 |
+
|
| 42 |
+
name: str
|
| 43 |
+
filters: List[Filter]
|
| 44 |
+
|
| 45 |
+
def apply(self, instances: List[Instance], docs: List[Dataset]) -> None:
|
| 46 |
+
resps = [inst.resps for inst in instances] # operate just on the model responses
|
| 47 |
+
for f in self.filters:
|
| 48 |
+
# apply filters in sequence
|
| 49 |
+
resps = f.apply(resps, docs)
|
| 50 |
+
|
| 51 |
+
# add the end results after filtering to filtered_requests of their respective source instances.
|
| 52 |
+
# has key `self.name`: each FilterEnsemble applied in a given run should use a different name.
|
| 53 |
+
for inst, resp in zip(instances, resps):
|
| 54 |
+
inst.filtered_resps[self.name] = resp
|
LaViDa/eval/lmms_eval/api/group.py
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import abc
|
| 2 |
+
from dataclasses import asdict, dataclass
|
| 3 |
+
from inspect import getsource
|
| 4 |
+
from typing import Any, Callable, List, Optional, Union
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
@dataclass
|
| 8 |
+
class AggMetricConfig(dict):
|
| 9 |
+
metric: Optional[str] = None
|
| 10 |
+
aggregation: Optional[str] = "mean"
|
| 11 |
+
weight_by_size: Optional[str] = False
|
| 12 |
+
# list of filter names which should be incorporated into the aggregated metric.
|
| 13 |
+
filter_list: Optional[Union[str, list]] = "none"
|
| 14 |
+
|
| 15 |
+
def __post_init__(self):
|
| 16 |
+
if self.aggregation != "mean" and not callable(self.aggregation):
|
| 17 |
+
raise ValueError(f"Currently, 'mean' is the only pre-defined aggregation across groups' subtasks. Got '{self.aggregation}'.")
|
| 18 |
+
|
| 19 |
+
if isinstance(self.filter_list, str):
|
| 20 |
+
self.filter_list = [self.filter_list]
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
@dataclass
|
| 24 |
+
class GroupConfig(dict):
|
| 25 |
+
group: Optional[str] = None
|
| 26 |
+
group_alias: Optional[str] = None
|
| 27 |
+
task: Optional[Union[str, list]] = None
|
| 28 |
+
aggregate_metric_list: Optional[Union[List[AggMetricConfig], AggMetricConfig, dict]] = None
|
| 29 |
+
metadata: Optional[dict] = None # by default, not used in the code. allows for users to pass arbitrary info to tasks
|
| 30 |
+
|
| 31 |
+
def __getitem__(self, item):
|
| 32 |
+
return getattr(self, item)
|
| 33 |
+
|
| 34 |
+
def __setitem__(self, item, value):
|
| 35 |
+
return setattr(self, item, value)
|
| 36 |
+
|
| 37 |
+
def __post_init__(self):
|
| 38 |
+
if self.aggregate_metric_list is not None:
|
| 39 |
+
if isinstance(self.aggregate_metric_list, dict):
|
| 40 |
+
self.aggregate_metric_list = [self.aggregate_metric_list]
|
| 41 |
+
|
| 42 |
+
self.aggregate_metric_list = [AggMetricConfig(**item) if isinstance(item, dict) else item for item in self.aggregate_metric_list]
|
| 43 |
+
|
| 44 |
+
def to_dict(self, keep_callable: bool = False) -> dict:
|
| 45 |
+
"""dumps the current config as a dictionary object, as a printable format.
|
| 46 |
+
null fields will not be printed.
|
| 47 |
+
Used for dumping results alongside full task configuration
|
| 48 |
+
|
| 49 |
+
:return: dict
|
| 50 |
+
A printable dictionary version of the TaskConfig object.
|
| 51 |
+
|
| 52 |
+
# TODO: should any default value in the TaskConfig not be printed?
|
| 53 |
+
"""
|
| 54 |
+
cfg_dict = asdict(self)
|
| 55 |
+
# remove values that are `None`
|
| 56 |
+
for k, v in list(cfg_dict.items()):
|
| 57 |
+
if callable(v):
|
| 58 |
+
cfg_dict[k] = self.serialize_function(v, keep_callable=keep_callable)
|
| 59 |
+
return cfg_dict
|
| 60 |
+
|
| 61 |
+
def serialize_function(self, value: Union[Callable, str], keep_callable=False) -> Union[Callable, str]:
|
| 62 |
+
"""Serializes a given function or string.
|
| 63 |
+
|
| 64 |
+
If 'keep_callable' is True, the original callable is returned.
|
| 65 |
+
Otherwise, attempts to return the source code of the callable using 'getsource'.
|
| 66 |
+
"""
|
| 67 |
+
if keep_callable:
|
| 68 |
+
return value
|
| 69 |
+
else:
|
| 70 |
+
try:
|
| 71 |
+
return getsource(value)
|
| 72 |
+
except (TypeError, OSError):
|
| 73 |
+
return str(value)
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
class ConfigurableGroup(abc.ABC):
|
| 77 |
+
def __init__(
|
| 78 |
+
self,
|
| 79 |
+
config: Optional[dict] = None,
|
| 80 |
+
) -> None:
|
| 81 |
+
self._config = GroupConfig(**config)
|
| 82 |
+
|
| 83 |
+
@property
|
| 84 |
+
def group(self):
|
| 85 |
+
return self._config.group
|
| 86 |
+
|
| 87 |
+
@property
|
| 88 |
+
def group_alias(self):
|
| 89 |
+
return self._config.group_alias
|
| 90 |
+
|
| 91 |
+
@property
|
| 92 |
+
def version(self):
|
| 93 |
+
return self._config.version
|
| 94 |
+
|
| 95 |
+
@property
|
| 96 |
+
def config(self):
|
| 97 |
+
return self._config.to_dict()
|
| 98 |
+
|
| 99 |
+
@property
|
| 100 |
+
def group_name(self) -> Any:
|
| 101 |
+
return self._config.group
|
| 102 |
+
|
| 103 |
+
def __repr__(self):
|
| 104 |
+
return f"ConfigurableGroup(group={self.group}," f"group_alias={self.group_alias})"
|
LaViDa/eval/lmms_eval/api/instance.py
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from dataclasses import dataclass, field
|
| 2 |
+
from typing import Literal, Tuple
|
| 3 |
+
|
| 4 |
+
|
| 5 |
+
@dataclass
|
| 6 |
+
class Instance:
|
| 7 |
+
request_type: Literal["loglikelihood", "generate_until", "generate_until_multi_round"]
|
| 8 |
+
arguments: tuple
|
| 9 |
+
idx: int
|
| 10 |
+
metadata: Tuple[str, int, int] = field(default_factory=lambda: (None, None, None)) # TODO: better typehints here
|
| 11 |
+
resps: list = field(default_factory=list)
|
| 12 |
+
filtered_resps: dict = field(default_factory=dict)
|
| 13 |
+
|
| 14 |
+
# initialized after init
|
| 15 |
+
task_name: str = None
|
| 16 |
+
doc_id: str = None
|
| 17 |
+
repeats: str = None
|
| 18 |
+
doc: dict = None
|
| 19 |
+
|
| 20 |
+
def __post_init__(self) -> None:
|
| 21 |
+
# unpack metadata field
|
| 22 |
+
self.task_name, self.doc_id, self.repeats = self.metadata["task"], self.metadata["doc_id"], self.metadata["repeats"]
|
| 23 |
+
|
| 24 |
+
@property
|
| 25 |
+
def args(self):
|
| 26 |
+
"""
|
| 27 |
+
Returns (string,) where `string` is the string to calculate loglikelihood over
|
| 28 |
+
"""
|
| 29 |
+
return self.arguments if isinstance(self.arguments, tuple) else (self.arguments,)
|
LaViDa/eval/lmms_eval/api/metrics.py
ADDED
|
@@ -0,0 +1,606 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# the code is adapted from https://github.com/EleutherAI/lm-evaluation-harness
|
| 2 |
+
import math
|
| 3 |
+
import random
|
| 4 |
+
import re
|
| 5 |
+
import string
|
| 6 |
+
from collections.abc import Iterable
|
| 7 |
+
from typing import List
|
| 8 |
+
|
| 9 |
+
import numpy as np
|
| 10 |
+
import sacrebleu
|
| 11 |
+
from loguru import logger as eval_logger
|
| 12 |
+
|
| 13 |
+
from lmms_eval.api.registry import register_aggregation, register_metric
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
# Register Aggregations First
|
| 17 |
+
@register_aggregation("bypass")
|
| 18 |
+
def bypass_agg(arr):
|
| 19 |
+
return 999
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
@register_aggregation("mean")
|
| 23 |
+
def mean(arr):
|
| 24 |
+
return sum(arr) / len(arr)
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
@register_aggregation("median")
|
| 28 |
+
def median(arr):
|
| 29 |
+
return arr[len(arr) // 2]
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
# Certain metrics must be calculated across all documents in a benchmark.
|
| 33 |
+
# We use them as aggregation metrics, paired with no-op passthrough metric fns.
|
| 34 |
+
@register_aggregation("perplexity")
|
| 35 |
+
def perplexity(items):
|
| 36 |
+
return math.exp(-mean(items))
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
@register_aggregation("weighted_perplexity")
|
| 40 |
+
def weighted_perplexity(items):
|
| 41 |
+
return math.exp(-weighted_mean(items))
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
@register_aggregation("bits_per_byte")
|
| 45 |
+
def bits_per_byte(items):
|
| 46 |
+
return -weighted_mean(items) / math.log(2)
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
@register_aggregation("f1")
|
| 50 |
+
def f1_score(items):
|
| 51 |
+
from sklearn.metrics import f1_score
|
| 52 |
+
|
| 53 |
+
unzipped_list = list(zip(*items))
|
| 54 |
+
golds = unzipped_list[0]
|
| 55 |
+
preds = unzipped_list[1]
|
| 56 |
+
fscore = f1_score(golds, preds)
|
| 57 |
+
|
| 58 |
+
return np.max(fscore)
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
@register_aggregation("matthews_corrcoef")
|
| 62 |
+
def matthews_corrcoef(items):
|
| 63 |
+
from sklearn.metrics import matthews_corrcoef
|
| 64 |
+
|
| 65 |
+
unzipped_list = list(zip(*items))
|
| 66 |
+
golds = unzipped_list[0]
|
| 67 |
+
preds = unzipped_list[1]
|
| 68 |
+
return matthews_corrcoef(golds, preds)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
@register_aggregation("bleu")
|
| 72 |
+
def bleu(items):
|
| 73 |
+
"""The Bilingual Evaluation Understudy Score, or BLEU for short, is a metric
|
| 74 |
+
for evaluating a generated sentence to a reference sentence. It counts matching
|
| 75 |
+
n-grams in the candidate translation to n-grams in the reference text, where
|
| 76 |
+
1-gram or unigram would be each token and a bigram comparison would be each
|
| 77 |
+
word pair. The comparison is made regardless of word order
|
| 78 |
+
Source: https://machinelearningmastery.com/calculate-bleu-score-for-text-python/
|
| 79 |
+
Paper: https://www.aclweb.org/anthology/P02-1040/
|
| 80 |
+
|
| 81 |
+
Higher is better
|
| 82 |
+
"""
|
| 83 |
+
refs = list(zip(*items))[0]
|
| 84 |
+
preds = list(zip(*items))[1]
|
| 85 |
+
refs, preds = _sacreformat(refs, preds)
|
| 86 |
+
return sacrebleu.corpus_bleu(preds, refs).score
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
@register_aggregation("chrf")
|
| 90 |
+
def chrf(items):
|
| 91 |
+
"""chrF++ is a tool for automatic evaluation of machine translation output
|
| 92 |
+
based on character n-gram precision and recall enhanced with word n-grams.
|
| 93 |
+
Source: https://github.com/m-popovic/chrF
|
| 94 |
+
Paper: https://www.aclweb.org/anthology/W15-3049.pdf
|
| 95 |
+
|
| 96 |
+
Higher is better # TODO I think
|
| 97 |
+
"""
|
| 98 |
+
refs = list(zip(*items))[0]
|
| 99 |
+
preds = list(zip(*items))[1]
|
| 100 |
+
refs, preds = _sacreformat(refs, preds)
|
| 101 |
+
return sacrebleu.corpus_chrf(preds, refs).score
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
@register_aggregation("ter")
|
| 105 |
+
def ter(items):
|
| 106 |
+
"""Translation Error Rate is an error metric for machine translation that
|
| 107 |
+
measures the number of edits required to change a system output into one
|
| 108 |
+
of the references
|
| 109 |
+
Source: http://www.cs.umd.edu/~snover/tercom/
|
| 110 |
+
Paper: http://mt-archive.info/AMTA-2006-Snover.pdf
|
| 111 |
+
|
| 112 |
+
Lower is better
|
| 113 |
+
"""
|
| 114 |
+
refs = list(zip(*items))[0]
|
| 115 |
+
preds = list(zip(*items))[1]
|
| 116 |
+
refs, preds = _sacreformat(refs, preds)
|
| 117 |
+
return sacrebleu.corpus_ter(preds, refs).score
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
@register_aggregation("brier_score")
|
| 121 |
+
def brier_score(items): # This is a passthrough function
|
| 122 |
+
gold, predictions = list(zip(*items))
|
| 123 |
+
bs, num_class = np.array(predictions).shape
|
| 124 |
+
|
| 125 |
+
gold = list(gold)
|
| 126 |
+
gold_one_hot = np.eye(num_class)[gold]
|
| 127 |
+
return np.mean(np.sum((predictions - gold_one_hot) ** 2, axis=1))
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
@register_metric(
|
| 131 |
+
metric="brier_score",
|
| 132 |
+
higher_is_better=False,
|
| 133 |
+
output_type=["multiple_choice"],
|
| 134 |
+
aggregation="brier_score",
|
| 135 |
+
)
|
| 136 |
+
def brier_score_fn(items): # This is a passthrough function
|
| 137 |
+
return items
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
@register_metric(
|
| 141 |
+
metric="acc",
|
| 142 |
+
higher_is_better=True,
|
| 143 |
+
output_type=["loglikelihood", "multiple_choice"],
|
| 144 |
+
aggregation="mean",
|
| 145 |
+
)
|
| 146 |
+
def acc_fn(items): # This is a passthrough function
|
| 147 |
+
return items
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
@register_metric(
|
| 151 |
+
metric="acc_norm",
|
| 152 |
+
higher_is_better=True,
|
| 153 |
+
output_type=["loglikelihood", "multiple_choice"],
|
| 154 |
+
aggregation="mean",
|
| 155 |
+
)
|
| 156 |
+
def acc_norm_fn(items): # This is a passthrough function
|
| 157 |
+
return items
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
@register_metric(
|
| 161 |
+
metric="acc_mutual_info",
|
| 162 |
+
higher_is_better=True,
|
| 163 |
+
output_type="multiple_choice",
|
| 164 |
+
aggregation="mean",
|
| 165 |
+
)
|
| 166 |
+
def acc_mutual_info_fn(items): # This is a passthrough function
|
| 167 |
+
return items
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
### the code used in the `exact_match_hf_evaluate` function is ported from
|
| 171 |
+
### https://github.com/huggingface/evaluate/blob/main/metrics/exact_match/exact_match.py
|
| 172 |
+
### which is under the apache license.
|
| 173 |
+
|
| 174 |
+
# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
|
| 175 |
+
|
| 176 |
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
| 177 |
+
# you may not use this file except in compliance with the License.
|
| 178 |
+
# You may obtain a copy of the License at
|
| 179 |
+
|
| 180 |
+
# http://www.apache.org/licenses/LICENSE-2.0
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
# Unless required by applicable law or agreed to in writing, software
|
| 184 |
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
| 185 |
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 186 |
+
# See the License for the specific language governing permissions and
|
| 187 |
+
# limitations under the License.
|
| 188 |
+
def exact_match_hf_evaluate(
|
| 189 |
+
predictions,
|
| 190 |
+
references,
|
| 191 |
+
regexes_to_ignore=None,
|
| 192 |
+
ignore_case=False,
|
| 193 |
+
ignore_punctuation=False,
|
| 194 |
+
ignore_numbers=False,
|
| 195 |
+
):
|
| 196 |
+
if regexes_to_ignore is not None:
|
| 197 |
+
for s in regexes_to_ignore:
|
| 198 |
+
predictions = np.array([re.sub(s, "", x) for x in predictions])
|
| 199 |
+
references = np.array([re.sub(s, "", x) for x in references])
|
| 200 |
+
else:
|
| 201 |
+
predictions = np.asarray(predictions)
|
| 202 |
+
references = np.asarray(references)
|
| 203 |
+
|
| 204 |
+
if ignore_case:
|
| 205 |
+
predictions = np.char.lower(predictions)
|
| 206 |
+
references = np.char.lower(references)
|
| 207 |
+
|
| 208 |
+
if ignore_punctuation:
|
| 209 |
+
repl_table = string.punctuation.maketrans("", "", string.punctuation)
|
| 210 |
+
predictions = np.char.translate(predictions, table=repl_table)
|
| 211 |
+
references = np.char.translate(references, table=repl_table)
|
| 212 |
+
|
| 213 |
+
if ignore_numbers:
|
| 214 |
+
repl_table = string.digits.maketrans("", "", string.digits)
|
| 215 |
+
predictions = np.char.translate(predictions, table=repl_table)
|
| 216 |
+
references = np.char.translate(references, table=repl_table)
|
| 217 |
+
|
| 218 |
+
score_list = predictions == references
|
| 219 |
+
|
| 220 |
+
return {"exact_match": np.mean(score_list)}
|
| 221 |
+
|
| 222 |
+
|
| 223 |
+
###
|
| 224 |
+
|
| 225 |
+
|
| 226 |
+
@register_metric(
|
| 227 |
+
metric="exact_match",
|
| 228 |
+
higher_is_better=True,
|
| 229 |
+
output_type="generate_until",
|
| 230 |
+
aggregation="mean",
|
| 231 |
+
)
|
| 232 |
+
def exact_match_fn(**kwargs):
|
| 233 |
+
return exact_match_hf_evaluate(**kwargs)
|
| 234 |
+
|
| 235 |
+
|
| 236 |
+
@register_metric(
|
| 237 |
+
metric="perplexity",
|
| 238 |
+
higher_is_better=False,
|
| 239 |
+
output_type="loglikelihood",
|
| 240 |
+
aggregation="perplexity",
|
| 241 |
+
)
|
| 242 |
+
def perplexity_fn(items): # This is a passthrough function
|
| 243 |
+
return items
|
| 244 |
+
|
| 245 |
+
|
| 246 |
+
@register_metric(
|
| 247 |
+
metric="word_perplexity",
|
| 248 |
+
higher_is_better=False,
|
| 249 |
+
output_type="loglikelihood_rolling",
|
| 250 |
+
aggregation="weighted_perplexity",
|
| 251 |
+
)
|
| 252 |
+
def word_perplexity_fn(items): # This is a passthrough function
|
| 253 |
+
return items
|
| 254 |
+
|
| 255 |
+
|
| 256 |
+
@register_metric(
|
| 257 |
+
metric="byte_perplexity",
|
| 258 |
+
higher_is_better=False,
|
| 259 |
+
output_type="loglikelihood_rolling",
|
| 260 |
+
aggregation="weighted_perplexity",
|
| 261 |
+
)
|
| 262 |
+
def byte_perplexity_fn(items): # This is a passthrough function
|
| 263 |
+
return items
|
| 264 |
+
|
| 265 |
+
|
| 266 |
+
@register_metric(
|
| 267 |
+
metric="bits_per_byte",
|
| 268 |
+
higher_is_better=False,
|
| 269 |
+
output_type="loglikelihood_rolling",
|
| 270 |
+
aggregation="bits_per_byte",
|
| 271 |
+
)
|
| 272 |
+
def bits_per_byte_fn(items): # This is a passthrough function
|
| 273 |
+
return items
|
| 274 |
+
|
| 275 |
+
|
| 276 |
+
def levenshtein_distance(s1, s2):
|
| 277 |
+
if len(s1) > len(s2):
|
| 278 |
+
s1, s2 = s2, s1
|
| 279 |
+
|
| 280 |
+
distances = range(len(s1) + 1)
|
| 281 |
+
for i2, c2 in enumerate(s2):
|
| 282 |
+
distances_ = [i2 + 1]
|
| 283 |
+
for i1, c1 in enumerate(s1):
|
| 284 |
+
if c1 == c2:
|
| 285 |
+
distances_.append(distances[i1])
|
| 286 |
+
else:
|
| 287 |
+
distances_.append(1 + min((distances[i1], distances[i1 + 1], distances_[-1])))
|
| 288 |
+
distances = distances_
|
| 289 |
+
return distances[-1]
|
| 290 |
+
|
| 291 |
+
|
| 292 |
+
@register_metric(
|
| 293 |
+
metric="anls",
|
| 294 |
+
higher_is_better=True,
|
| 295 |
+
output_type="generate_until",
|
| 296 |
+
aggregation="mean",
|
| 297 |
+
)
|
| 298 |
+
def anls(
|
| 299 |
+
references,
|
| 300 |
+
predictions,
|
| 301 |
+
thresh_hold=0.5,
|
| 302 |
+
):
|
| 303 |
+
"""https://github.com/QwenLM/Qwen-VL/blob/master/eval_mm/infographicsvqa_eval.py"""
|
| 304 |
+
values = []
|
| 305 |
+
# Unwrap predictions if it's a nested list
|
| 306 |
+
pred = predictions[0] if isinstance(predictions[0], str) else predictions[0][0]
|
| 307 |
+
|
| 308 |
+
for answer in references:
|
| 309 |
+
# preprocess both the answers - gt and prediction
|
| 310 |
+
gt_answer = " ".join(answer.strip().lower().split())
|
| 311 |
+
det_answer = " ".join(pred.strip().lower().split())
|
| 312 |
+
|
| 313 |
+
dist = levenshtein_distance(gt_answer, det_answer)
|
| 314 |
+
length = max(len(answer.upper()), len(pred.upper()))
|
| 315 |
+
values.append(0.0 if length == 0 else float(dist) / float(length))
|
| 316 |
+
|
| 317 |
+
question_result = 1 - min(values)
|
| 318 |
+
|
| 319 |
+
if question_result < thresh_hold:
|
| 320 |
+
question_result = 0
|
| 321 |
+
return {"anls": question_result}
|
| 322 |
+
|
| 323 |
+
|
| 324 |
+
def pop_stddev(arr):
|
| 325 |
+
mu = mean(arr)
|
| 326 |
+
return math.sqrt(sum([(x - mu) ** 2 for x in arr]) / len(arr))
|
| 327 |
+
|
| 328 |
+
|
| 329 |
+
def sample_stddev(arr):
|
| 330 |
+
mu = mean(arr)
|
| 331 |
+
return math.sqrt(sum([(x - mu) ** 2 for x in arr]) / (len(arr) - 1))
|
| 332 |
+
|
| 333 |
+
|
| 334 |
+
def mean_stderr(arr):
|
| 335 |
+
return sample_stddev(arr) / math.sqrt(len(arr))
|
| 336 |
+
|
| 337 |
+
|
| 338 |
+
@register_metric(
|
| 339 |
+
metric="bypass",
|
| 340 |
+
higher_is_better=True,
|
| 341 |
+
output_type=["loglikelihood", "multiple_choice", "generate_until", "generate_until_multi_round"],
|
| 342 |
+
aggregation="bypass",
|
| 343 |
+
)
|
| 344 |
+
def bypass(items):
|
| 345 |
+
return items
|
| 346 |
+
|
| 347 |
+
|
| 348 |
+
@register_metric(
|
| 349 |
+
metric="mcc",
|
| 350 |
+
higher_is_better=True,
|
| 351 |
+
output_type="multiple_choice",
|
| 352 |
+
aggregation="matthews_corrcoef",
|
| 353 |
+
)
|
| 354 |
+
def mcc_fn(items): # This is a passthrough function
|
| 355 |
+
return items
|
| 356 |
+
|
| 357 |
+
|
| 358 |
+
@register_metric(
|
| 359 |
+
metric="f1",
|
| 360 |
+
higher_is_better=True,
|
| 361 |
+
output_type="multiple_choice",
|
| 362 |
+
aggregation="f1",
|
| 363 |
+
)
|
| 364 |
+
def f1_fn(items): # This is a passthrough function
|
| 365 |
+
return items
|
| 366 |
+
|
| 367 |
+
|
| 368 |
+
@register_metric(
|
| 369 |
+
metric="bleu",
|
| 370 |
+
higher_is_better=True,
|
| 371 |
+
output_type=["generate_until", "generate_until_multi_round"],
|
| 372 |
+
aggregation="bleu",
|
| 373 |
+
)
|
| 374 |
+
def bleu_fn(items): # This is a passthrough function
|
| 375 |
+
return items
|
| 376 |
+
|
| 377 |
+
|
| 378 |
+
@register_metric(
|
| 379 |
+
metric="chrf",
|
| 380 |
+
higher_is_better=True,
|
| 381 |
+
output_type=["generate_until", "generate_until_multi_round"],
|
| 382 |
+
aggregation="chrf",
|
| 383 |
+
)
|
| 384 |
+
def chrf_fn(items): # This is a passthrough function
|
| 385 |
+
return items
|
| 386 |
+
|
| 387 |
+
|
| 388 |
+
@register_metric(
|
| 389 |
+
metric="ter",
|
| 390 |
+
higher_is_better=True,
|
| 391 |
+
output_type=["generate_until", "generate_until_multi_round"],
|
| 392 |
+
aggregation="ter",
|
| 393 |
+
)
|
| 394 |
+
def ter_fn(items): # This is a passthrough function
|
| 395 |
+
return items
|
| 396 |
+
|
| 397 |
+
|
| 398 |
+
@register_metric(
|
| 399 |
+
metric="acc_all",
|
| 400 |
+
higher_is_better=True,
|
| 401 |
+
output_type="loglikelihood",
|
| 402 |
+
aggregation="mean",
|
| 403 |
+
)
|
| 404 |
+
def acc_all(items):
|
| 405 |
+
# Only count as correct if all answers are labeled correctly for each question
|
| 406 |
+
question_scoring_dict = {}
|
| 407 |
+
preds = list(zip(*items))[0]
|
| 408 |
+
docs = list(zip(*items))[1]
|
| 409 |
+
|
| 410 |
+
for doc, pred in zip(docs, preds):
|
| 411 |
+
paragraph_id = doc["idx"]["paragraph"]
|
| 412 |
+
question_id = doc["idx"]["question"]
|
| 413 |
+
if (paragraph_id, question_id) not in question_scoring_dict:
|
| 414 |
+
question_scoring_dict[(paragraph_id, question_id)] = []
|
| 415 |
+
|
| 416 |
+
gold_label = doc["label"] == 1
|
| 417 |
+
|
| 418 |
+
question_scoring_dict[(paragraph_id, question_id)].append(gold_label == pred)
|
| 419 |
+
acc = np.mean([int(all(x)) for x in question_scoring_dict.values()])
|
| 420 |
+
return acc
|
| 421 |
+
|
| 422 |
+
|
| 423 |
+
def acc_all_stderr(items):
|
| 424 |
+
# Only count as correct if all answers are labeled correctly for each question
|
| 425 |
+
question_scoring_dict = {}
|
| 426 |
+
preds = list(zip(*items))[0]
|
| 427 |
+
docs = list(zip(*items))[1]
|
| 428 |
+
|
| 429 |
+
for doc, pred in zip(docs, preds):
|
| 430 |
+
question_id = doc["idx"]["question"]
|
| 431 |
+
if question_id not in question_scoring_dict:
|
| 432 |
+
question_scoring_dict[question_id] = []
|
| 433 |
+
|
| 434 |
+
gold_label = doc["label"] == 1
|
| 435 |
+
question_scoring_dict[question_id].append(gold_label == pred)
|
| 436 |
+
|
| 437 |
+
acc = mean_stderr([int(all(x)) for x in question_scoring_dict.values()])
|
| 438 |
+
return acc
|
| 439 |
+
|
| 440 |
+
|
| 441 |
+
def metric_max_over_ground_truths(metric_fn, prediction, ground_truths):
|
| 442 |
+
"""Compute max metric between prediction and each ground truth."""
|
| 443 |
+
scores_for_ground_truths = []
|
| 444 |
+
for ground_truth in ground_truths:
|
| 445 |
+
score = metric_fn(prediction, ground_truth)
|
| 446 |
+
scores_for_ground_truths.append(score)
|
| 447 |
+
return max(scores_for_ground_truths)
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
def weighted_mean(items):
|
| 451 |
+
a, b = zip(*items)
|
| 452 |
+
return sum(a) / sum(b)
|
| 453 |
+
|
| 454 |
+
|
| 455 |
+
def is_non_str_iterable(obj):
|
| 456 |
+
return isinstance(obj, Iterable) and not isinstance(obj, str)
|
| 457 |
+
|
| 458 |
+
|
| 459 |
+
def _sacreformat(refs, preds):
|
| 460 |
+
"""Format refs and preds for sacrebleu corpus calculation. It is very particular"""
|
| 461 |
+
# Sacrebleu expects (List[str], List[List[str])
|
| 462 |
+
# e.g. sacrebleu.corpus_bleu([pred_t], [[ref1_stream], [ref2_stream], ...])
|
| 463 |
+
|
| 464 |
+
# Note [ref1_stream] is the first reference for each pred.
|
| 465 |
+
# So lists are size N and (M, N) for N preds and M possible refs for each pred
|
| 466 |
+
# This is a different order of dimensions that I would expect
|
| 467 |
+
|
| 468 |
+
# We expect refs to be List[str] or List[List[str]], the outer list corresponding to preds
|
| 469 |
+
# Must become List[List[str]] with the inner list corresponding to preds
|
| 470 |
+
if not is_non_str_iterable(refs):
|
| 471 |
+
refs = list(refs)
|
| 472 |
+
if not is_non_str_iterable(refs[0]):
|
| 473 |
+
refs = [[ref] for ref in refs]
|
| 474 |
+
refs = list(zip(*refs))
|
| 475 |
+
# Note the number of refs in each ref list much match the number of preds
|
| 476 |
+
|
| 477 |
+
# We expect preds to be List[str] or List[List[str]]. Must become List[str]
|
| 478 |
+
if not is_non_str_iterable(preds):
|
| 479 |
+
preds = list(preds)
|
| 480 |
+
if is_non_str_iterable(preds[0]):
|
| 481 |
+
assert len(preds[0]) == 1, f"Pred must be a str, was {preds[0]}"
|
| 482 |
+
preds = [pred[0] for pred in preds]
|
| 483 |
+
|
| 484 |
+
return refs, preds
|
| 485 |
+
|
| 486 |
+
|
| 487 |
+
# stderr stuff
|
| 488 |
+
|
| 489 |
+
|
| 490 |
+
class _bootstrap_internal:
|
| 491 |
+
def __init__(self, f, n) -> None:
|
| 492 |
+
self.f = f
|
| 493 |
+
self.n = n
|
| 494 |
+
|
| 495 |
+
def __call__(self, v):
|
| 496 |
+
i, xs = v
|
| 497 |
+
rnd = random.Random()
|
| 498 |
+
rnd.seed(i)
|
| 499 |
+
res = []
|
| 500 |
+
for _ in range(self.n):
|
| 501 |
+
res.append(self.f(rnd.choices(xs, k=len(xs))))
|
| 502 |
+
return res
|
| 503 |
+
|
| 504 |
+
|
| 505 |
+
def bootstrap_stderr(f, xs, iters):
|
| 506 |
+
import multiprocessing as mp
|
| 507 |
+
|
| 508 |
+
pool = mp.Pool(mp.cpu_count())
|
| 509 |
+
# this gives a biased estimate of the stderr (i.e w/ the mean, it gives something
|
| 510 |
+
# equivalent to stderr calculated without Bessel's correction in the stddev.
|
| 511 |
+
# Unfortunately, I haven't been able to figure out what the right correction is
|
| 512 |
+
# to make the bootstrap unbiased - i considered multiplying by sqrt(n/(n-1)) but
|
| 513 |
+
# that would be ad-hoc and I can't prove that that would actually be an unbiased estimator)
|
| 514 |
+
# Thankfully, shouldn't matter because our samples are pretty big usually anyways
|
| 515 |
+
res = []
|
| 516 |
+
chunk_size = min(1000, iters)
|
| 517 |
+
from tqdm import tqdm
|
| 518 |
+
|
| 519 |
+
print("bootstrapping for stddev:", f.__name__)
|
| 520 |
+
for bootstrap in tqdm(
|
| 521 |
+
pool.imap(
|
| 522 |
+
_bootstrap_internal(f, chunk_size),
|
| 523 |
+
[(i, xs) for i in range(iters // chunk_size)],
|
| 524 |
+
),
|
| 525 |
+
total=iters // chunk_size,
|
| 526 |
+
):
|
| 527 |
+
# sample w replacement
|
| 528 |
+
res.extend(bootstrap)
|
| 529 |
+
|
| 530 |
+
pool.close()
|
| 531 |
+
return sample_stddev(res)
|
| 532 |
+
|
| 533 |
+
|
| 534 |
+
def stderr_for_metric(metric, bootstrap_iters: int):
|
| 535 |
+
if bootstrap_iters <= 0:
|
| 536 |
+
# return no function (don't compute stderr) if bootstrap iters = 0
|
| 537 |
+
return None
|
| 538 |
+
|
| 539 |
+
bootstrappable = [
|
| 540 |
+
median,
|
| 541 |
+
matthews_corrcoef,
|
| 542 |
+
f1_score,
|
| 543 |
+
perplexity,
|
| 544 |
+
bleu,
|
| 545 |
+
chrf,
|
| 546 |
+
ter,
|
| 547 |
+
]
|
| 548 |
+
|
| 549 |
+
if metric in bootstrappable:
|
| 550 |
+
return lambda x: bootstrap_stderr(metric, x, iters=bootstrap_iters)
|
| 551 |
+
|
| 552 |
+
stderr = {mean: mean_stderr, acc_all: acc_all_stderr}
|
| 553 |
+
|
| 554 |
+
return stderr.get(metric, None)
|
| 555 |
+
|
| 556 |
+
|
| 557 |
+
def pooled_sample_stderr(stderrs: List[float], sizes: List[int]):
|
| 558 |
+
# Used to aggregate bootstrapped stderrs across subtasks in a group,
|
| 559 |
+
# when we are weighting by the size of each subtask.
|
| 560 |
+
#
|
| 561 |
+
|
| 562 |
+
assert len(stderrs) == len(sizes)
|
| 563 |
+
|
| 564 |
+
# formula source: https://en.wikipedia.org/wiki/Pooled_variance
|
| 565 |
+
# and: https://stats.stackexchange.com/a/4841331
|
| 566 |
+
# this empirically seems to match running `stderr_for_metric` on all instances
|
| 567 |
+
# from the subtasks concatenated with each other.
|
| 568 |
+
pooled_sample_var = (sum([(size - 1) * stderr**2 * size for size, stderr in zip(sizes, stderrs)])) / (sum(sizes) - len(sizes))
|
| 569 |
+
|
| 570 |
+
return np.sqrt(pooled_sample_var / sum(sizes))
|
| 571 |
+
|
| 572 |
+
|
| 573 |
+
def combined_sample_stderr(stderrs: List[float], sizes: List[int], metrics=None):
|
| 574 |
+
assert metrics is not None, "Need to pass a list of each subtask's metric for this stderr aggregation"
|
| 575 |
+
assert len(stderrs) == len(sizes) and len(sizes) == len(metrics)
|
| 576 |
+
|
| 577 |
+
# See https://github.com/EleutherAI/lm-evaluation-harness/pull/1390 for more documentation.
|
| 578 |
+
# This formula depends on sample means.
|
| 579 |
+
# removed because it seems to give erroneously huge stderrs for groupings of tasks
|
| 580 |
+
# and does not seem to match up with bootstrap-calculated stderrs for groups.
|
| 581 |
+
|
| 582 |
+
### don't use this unless a statistician has told you it's the right thing to do ###
|
| 583 |
+
|
| 584 |
+
# accumulators: we'll aggregate pairwise N - 1 times
|
| 585 |
+
variance = stderrs[0] ** 2
|
| 586 |
+
curr_size = sizes[0]
|
| 587 |
+
curr_score = metrics[0]
|
| 588 |
+
|
| 589 |
+
for stderr, size, score in zip(stderrs[1:], sizes[1:], metrics[1:]):
|
| 590 |
+
curr_score = ((curr_score * curr_size) + (score * size)) / (curr_size + size) # NOTE: this assumes our aggregation fn is "mean"
|
| 591 |
+
|
| 592 |
+
variance = ((curr_size - 1) * variance + (size - 1) * (stderr**2)) / (curr_size + size - 1) + curr_size * size / ((curr_size + size) * (curr_size + size - 1)) * (curr_score - score) ** 2
|
| 593 |
+
|
| 594 |
+
return np.sqrt(variance)
|
| 595 |
+
|
| 596 |
+
|
| 597 |
+
def aggregate_subtask_metrics(metrics, sizes, weight_by_size=True):
|
| 598 |
+
# A helper function that is used to aggregate
|
| 599 |
+
# subtask scores cross-task.
|
| 600 |
+
# TODO: does not hold for non-mean aggregations
|
| 601 |
+
if not weight_by_size:
|
| 602 |
+
sizes = [1] * len(sizes)
|
| 603 |
+
|
| 604 |
+
assert len(metrics) == len(sizes)
|
| 605 |
+
|
| 606 |
+
return sum([metric * size for metric, size in zip(metrics, sizes)]) / sum(sizes)
|
LaViDa/eval/lmms_eval/api/model.py
ADDED
|
@@ -0,0 +1,221 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import abc
|
| 2 |
+
import hashlib
|
| 3 |
+
import json
|
| 4 |
+
import os
|
| 5 |
+
from typing import List, Optional, Tuple, Type, TypeVar, Union
|
| 6 |
+
|
| 7 |
+
from loguru import logger as eval_logger
|
| 8 |
+
from sqlitedict import SqliteDict
|
| 9 |
+
from tqdm import tqdm
|
| 10 |
+
|
| 11 |
+
from lmms_eval import utils
|
| 12 |
+
from lmms_eval.api.instance import Instance
|
| 13 |
+
|
| 14 |
+
T = TypeVar("T", bound="lmms")
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
class lmms(abc.ABC):
|
| 18 |
+
def __init__(self) -> None:
|
| 19 |
+
"""Defines the interface that should be implemented by all lmms subclasses.
|
| 20 |
+
lmmss are assumed to take image-text as input and yield strings as output
|
| 21 |
+
(inputs/outputs should be tokenization-agnostic.)
|
| 22 |
+
"""
|
| 23 |
+
# set rank and world size to a single process, by default.
|
| 24 |
+
self._rank = 0
|
| 25 |
+
self._world_size = 1
|
| 26 |
+
self.cache_hook = CacheHook(None)
|
| 27 |
+
self.task_dict = {}
|
| 28 |
+
|
| 29 |
+
@abc.abstractmethod
|
| 30 |
+
def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
|
| 31 |
+
"""Compute log-likelihood of generating a continuation from a context.
|
| 32 |
+
Downstream tasks should attempt to use loglikelihood instead of other
|
| 33 |
+
LMM calls whenever possible.
|
| 34 |
+
|
| 35 |
+
:param requests: list[Instance]
|
| 36 |
+
A list of Instance objects, with property `args` which returns a tuple (context, continuation).
|
| 37 |
+
`context: str`
|
| 38 |
+
Context string. Implementations of LMM must be able to handle an
|
| 39 |
+
empty context string.
|
| 40 |
+
`continuation: str`
|
| 41 |
+
The continuation over which log likelihood will be calculated. If
|
| 42 |
+
there is a word boundary, the space should be in the continuation.
|
| 43 |
+
For example, context="hello" continuation=" world" is correct.
|
| 44 |
+
'visual_list: list[dict]'
|
| 45 |
+
Visual input to the model. Can be None.
|
| 46 |
+
|
| 47 |
+
:return: list[tuple[float, bool]]
|
| 48 |
+
A list of pairs (logprob, isgreedy)
|
| 49 |
+
`logprob: float`
|
| 50 |
+
The log probability of `continuation`.
|
| 51 |
+
`isgreedy`:
|
| 52 |
+
Whether `continuation` would be generated by greedy sampling from `context`.
|
| 53 |
+
"""
|
| 54 |
+
pass
|
| 55 |
+
|
| 56 |
+
# TODO: Add an optional max length
|
| 57 |
+
@abc.abstractmethod
|
| 58 |
+
def generate_until(self, requests) -> List[str]:
|
| 59 |
+
"""Generate greedily until a stopping sequence
|
| 60 |
+
|
| 61 |
+
:param requests: list[Instance]
|
| 62 |
+
A list of Instance objects with property `args` which returns a tuple (context, until).
|
| 63 |
+
context: str
|
| 64 |
+
Context string
|
| 65 |
+
generation_kwargs: dict
|
| 66 |
+
Generation Kwargs
|
| 67 |
+
'visual_list: list[dict]'
|
| 68 |
+
Visual input to the model. Can be None.
|
| 69 |
+
:return: list[str]
|
| 70 |
+
A list of strings continuation
|
| 71 |
+
continuation: str
|
| 72 |
+
The generated continuation.
|
| 73 |
+
"""
|
| 74 |
+
pass
|
| 75 |
+
|
| 76 |
+
@abc.abstractmethod
|
| 77 |
+
def generate_until_multi_round(self, requests) -> List[str]:
|
| 78 |
+
"""Generate greedily until a stopping sequence
|
| 79 |
+
|
| 80 |
+
:param requests: list[Instance]
|
| 81 |
+
A list of Instance objects with property `args` which returns a tuple (context, until).
|
| 82 |
+
context: str
|
| 83 |
+
Context string
|
| 84 |
+
generation_kwargs: dict
|
| 85 |
+
Generation Kwargs
|
| 86 |
+
'visual_list: list[dict]'
|
| 87 |
+
Visual input to the model. Can be None.
|
| 88 |
+
:return: list[str]
|
| 89 |
+
A list of strings continuation
|
| 90 |
+
continuation: str
|
| 91 |
+
The generated continuation.
|
| 92 |
+
"""
|
| 93 |
+
pass
|
| 94 |
+
|
| 95 |
+
@classmethod
|
| 96 |
+
def create_from_arg_string(cls: Type[T], arg_string: str, additional_config: Optional[dict] = None) -> T:
|
| 97 |
+
"""
|
| 98 |
+
Creates an instance of the LMM class using the given argument string and additional config.
|
| 99 |
+
|
| 100 |
+
Parameters:
|
| 101 |
+
- arg_string: A string containing arguments in the format key1=value1,key2=value2.
|
| 102 |
+
- additional_config: Optional dictionary containing additional configuration parameters.
|
| 103 |
+
|
| 104 |
+
Returns:
|
| 105 |
+
- Instance of the LMM class.
|
| 106 |
+
"""
|
| 107 |
+
additional_config = {} if additional_config is None else additional_config
|
| 108 |
+
args = utils.simple_parse_args_string(arg_string)
|
| 109 |
+
args2 = {k: v for k, v in additional_config.items() if v is not None}
|
| 110 |
+
return cls(**args, **args2)
|
| 111 |
+
|
| 112 |
+
@property
|
| 113 |
+
def rank(self):
|
| 114 |
+
# used in the case of parallelism. Hardcoded to
|
| 115 |
+
# ensure no errors arise using API models which do
|
| 116 |
+
# not support multi-device parallelism nor expect it.
|
| 117 |
+
return self._rank
|
| 118 |
+
|
| 119 |
+
@property
|
| 120 |
+
def world_size(self):
|
| 121 |
+
# used in the case of parallelism. Hardcoded to
|
| 122 |
+
# ensure no errors arise using API models which do
|
| 123 |
+
# not support multi-device parallelism nor expect it.
|
| 124 |
+
return self._world_size
|
| 125 |
+
|
| 126 |
+
def set_cache_hook(self, cache_hook) -> None:
|
| 127 |
+
self.cache_hook = cache_hook
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
### SQLite-based caching of LMM responses
|
| 131 |
+
def hash_args(attr, args):
|
| 132 |
+
dat = json.dumps([attr] + list(args))
|
| 133 |
+
return hashlib.sha256(dat.encode("utf-8")).hexdigest()
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
class CacheHook:
|
| 137 |
+
def __init__(self, cachinglm) -> None:
|
| 138 |
+
if cachinglm is None:
|
| 139 |
+
self.dbdict = None
|
| 140 |
+
return
|
| 141 |
+
|
| 142 |
+
self.dbdict = cachinglm.dbdict
|
| 143 |
+
|
| 144 |
+
def add_partial(self, attr, req, res) -> None:
|
| 145 |
+
if self.dbdict is None:
|
| 146 |
+
return
|
| 147 |
+
hsh = hash_args(attr, req)
|
| 148 |
+
self.dbdict[hsh] = res
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
class CachingLMM:
|
| 152 |
+
def __init__(self, lm, cache_db) -> None:
|
| 153 |
+
"""LMM wrapper that returns cached results if they exist, and uses the underlying LMM if not.
|
| 154 |
+
|
| 155 |
+
:param lm: LMM
|
| 156 |
+
Underlying LMM
|
| 157 |
+
:param cache_db: str
|
| 158 |
+
Path to cache db
|
| 159 |
+
"""
|
| 160 |
+
self.lm = lm
|
| 161 |
+
self.cache_db = cache_db
|
| 162 |
+
if os.path.dirname(cache_db):
|
| 163 |
+
os.makedirs(os.path.dirname(cache_db), exist_ok=True)
|
| 164 |
+
self.dbdict = SqliteDict(cache_db, autocommit=True)
|
| 165 |
+
|
| 166 |
+
# add hook to lm
|
| 167 |
+
lm.set_cache_hook(self.get_cache_hook())
|
| 168 |
+
|
| 169 |
+
def __getattr__(self, attr):
|
| 170 |
+
lm_attr = getattr(self.lm, attr)
|
| 171 |
+
if not callable(lm_attr):
|
| 172 |
+
return lm_attr
|
| 173 |
+
|
| 174 |
+
def fn(requests):
|
| 175 |
+
res = []
|
| 176 |
+
remaining_reqs = []
|
| 177 |
+
warned = False
|
| 178 |
+
# figure out which ones are cached and which ones are new
|
| 179 |
+
eval_logger.info(f"Loading '{attr}' responses from cache '{self.cache_db}' where possible...")
|
| 180 |
+
for req in tqdm(requests):
|
| 181 |
+
hsh = hash_args(attr, req.args)
|
| 182 |
+
if attr in ["generate_until", "generate_until_multi_round"] and req.args[1].get("do_sample", False):
|
| 183 |
+
# when we are doing non-greedy generation, don't use the cache
|
| 184 |
+
# (else every "randomly sampled" generation would be identical for repeats > 1).
|
| 185 |
+
if not warned:
|
| 186 |
+
eval_logger.warning(f"Arguments to lm.generate_until() '{req.args[1]}' include non-deterministic sampling. Caching will not be performed for such requests.")
|
| 187 |
+
warned = True
|
| 188 |
+
res.append(None)
|
| 189 |
+
remaining_reqs.append(req)
|
| 190 |
+
elif hsh in self.dbdict:
|
| 191 |
+
ob = self.dbdict[hsh]
|
| 192 |
+
|
| 193 |
+
assert ob is not None
|
| 194 |
+
|
| 195 |
+
res.append(ob)
|
| 196 |
+
else:
|
| 197 |
+
res.append(None)
|
| 198 |
+
remaining_reqs.append(req)
|
| 199 |
+
|
| 200 |
+
# actually run the LMM on the requests that do not have cached results
|
| 201 |
+
rem_res = getattr(self.lm, attr)(remaining_reqs)
|
| 202 |
+
|
| 203 |
+
# stick the new ones back into the list and also cache any of the new ones
|
| 204 |
+
resptr = 0
|
| 205 |
+
for req, r in zip(remaining_reqs, rem_res):
|
| 206 |
+
while res[resptr] is not None:
|
| 207 |
+
resptr += 1
|
| 208 |
+
|
| 209 |
+
res[resptr] = r
|
| 210 |
+
|
| 211 |
+
# caching
|
| 212 |
+
hsh = hash_args(attr, req.args)
|
| 213 |
+
self.dbdict[hsh] = r
|
| 214 |
+
self.dbdict.commit()
|
| 215 |
+
|
| 216 |
+
return res
|
| 217 |
+
|
| 218 |
+
return fn
|
| 219 |
+
|
| 220 |
+
def get_cache_hook(self):
|
| 221 |
+
return CacheHook(self)
|
LaViDa/eval/lmms_eval/api/registry.py
ADDED
|
@@ -0,0 +1,185 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from typing import Callable, Dict, Union
|
| 2 |
+
|
| 3 |
+
import evaluate as hf_evaluate
|
| 4 |
+
from loguru import logger as eval_logger
|
| 5 |
+
|
| 6 |
+
from lmms_eval.api.model import lmms
|
| 7 |
+
|
| 8 |
+
MODEL_REGISTRY = {}
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
def register_model(*names):
|
| 12 |
+
# either pass a list or a single alias.
|
| 13 |
+
# function receives them as a tuple of strings
|
| 14 |
+
|
| 15 |
+
def decorate(cls):
|
| 16 |
+
for name in names:
|
| 17 |
+
assert issubclass(cls, lmms), f"Model '{name}' ({cls.__name__}) must extend lmms class"
|
| 18 |
+
|
| 19 |
+
assert name not in MODEL_REGISTRY, f"Model named '{name}' conflicts with existing model! Please register with a non-conflicting alias instead."
|
| 20 |
+
|
| 21 |
+
MODEL_REGISTRY[name] = cls
|
| 22 |
+
return cls
|
| 23 |
+
|
| 24 |
+
return decorate
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def get_model(model_name):
|
| 28 |
+
try:
|
| 29 |
+
return MODEL_REGISTRY[model_name]
|
| 30 |
+
except KeyError:
|
| 31 |
+
raise ValueError(f"Attempted to load model '{model_name}', but no model for this name found! Supported model names: {', '.join(MODEL_REGISTRY.keys())}")
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
TASK_REGISTRY = {} # Key: task name, Value: task ConfigurableTask class
|
| 35 |
+
GROUP_REGISTRY = {} # Key: group name, Value: list of task names or group names
|
| 36 |
+
TASK_INITIALIZED = False
|
| 37 |
+
ALL_TASKS = set() # Set of all task names and group names
|
| 38 |
+
func2task_index = {} # Key: task ConfigurableTask class, Value: task name
|
| 39 |
+
OUTPUT_TYPE_REGISTRY = {}
|
| 40 |
+
METRIC_REGISTRY = {}
|
| 41 |
+
METRIC_AGGREGATION_REGISTRY = {}
|
| 42 |
+
AGGREGATION_REGISTRY: Dict[str, Callable[[], Dict[str, Callable]]] = {}
|
| 43 |
+
HIGHER_IS_BETTER_REGISTRY = {}
|
| 44 |
+
FILTER_REGISTRY = {}
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def register_task(name):
|
| 48 |
+
def decorate(fn):
|
| 49 |
+
assert name not in TASK_REGISTRY, f"task named '{name}' conflicts with existing registered task!"
|
| 50 |
+
|
| 51 |
+
TASK_REGISTRY[name] = fn
|
| 52 |
+
ALL_TASKS.add(name)
|
| 53 |
+
func2task_index[fn.__name__] = name
|
| 54 |
+
return fn
|
| 55 |
+
|
| 56 |
+
return decorate
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def register_group(name):
|
| 60 |
+
def decorate(fn):
|
| 61 |
+
func_name = func2task_index[fn.__name__]
|
| 62 |
+
if name in GROUP_REGISTRY:
|
| 63 |
+
GROUP_REGISTRY[name].append(func_name)
|
| 64 |
+
else:
|
| 65 |
+
GROUP_REGISTRY[name] = [func_name]
|
| 66 |
+
ALL_TASKS.add(name)
|
| 67 |
+
return fn
|
| 68 |
+
|
| 69 |
+
return decorate
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
OUTPUT_TYPE_REGISTRY = {}
|
| 73 |
+
METRIC_REGISTRY = {}
|
| 74 |
+
METRIC_AGGREGATION_REGISTRY = {}
|
| 75 |
+
AGGREGATION_REGISTRY = {}
|
| 76 |
+
HIGHER_IS_BETTER_REGISTRY = {}
|
| 77 |
+
|
| 78 |
+
DEFAULT_METRIC_REGISTRY = {
|
| 79 |
+
"loglikelihood": [
|
| 80 |
+
"perplexity",
|
| 81 |
+
"acc",
|
| 82 |
+
],
|
| 83 |
+
"multiple_choice": ["acc", "acc_norm"],
|
| 84 |
+
"generate_until": ["exact_match"],
|
| 85 |
+
"generate_until_multi_round": ["exact_match"],
|
| 86 |
+
}
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def register_metric(**args):
|
| 90 |
+
# TODO: do we want to enforce a certain interface to registered metrics?
|
| 91 |
+
def decorate(fn):
|
| 92 |
+
assert "metric" in args
|
| 93 |
+
name = args["metric"]
|
| 94 |
+
|
| 95 |
+
for key, registry in [
|
| 96 |
+
("metric", METRIC_REGISTRY),
|
| 97 |
+
("higher_is_better", HIGHER_IS_BETTER_REGISTRY),
|
| 98 |
+
("aggregation", METRIC_AGGREGATION_REGISTRY),
|
| 99 |
+
]:
|
| 100 |
+
if key in args:
|
| 101 |
+
value = args[key]
|
| 102 |
+
assert value not in registry, f"{key} named '{value}' conflicts with existing registered {key}!"
|
| 103 |
+
|
| 104 |
+
if key == "metric":
|
| 105 |
+
registry[name] = fn
|
| 106 |
+
elif key == "aggregation":
|
| 107 |
+
registry[name] = AGGREGATION_REGISTRY[value]
|
| 108 |
+
else:
|
| 109 |
+
registry[name] = value
|
| 110 |
+
|
| 111 |
+
return fn
|
| 112 |
+
|
| 113 |
+
return decorate
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
def get_metric(name: str, hf_evaluate_metric=False) -> Callable:
|
| 117 |
+
if not hf_evaluate_metric:
|
| 118 |
+
if name in METRIC_REGISTRY:
|
| 119 |
+
return METRIC_REGISTRY[name]
|
| 120 |
+
else:
|
| 121 |
+
eval_logger.warning(f"Could not find registered metric '{name}' in lm-eval, searching in HF Evaluate library...")
|
| 122 |
+
|
| 123 |
+
try:
|
| 124 |
+
metric_object = hf_evaluate.load(name)
|
| 125 |
+
return metric_object.compute
|
| 126 |
+
except Exception:
|
| 127 |
+
eval_logger.error(
|
| 128 |
+
f"{name} not found in the evaluate library! Please check https://huggingface.co/evaluate-metric",
|
| 129 |
+
)
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def register_aggregation(name):
|
| 133 |
+
def decorate(fn):
|
| 134 |
+
assert name not in AGGREGATION_REGISTRY, f"aggregation named '{name}' conflicts with existing registered aggregation!"
|
| 135 |
+
|
| 136 |
+
AGGREGATION_REGISTRY[name] = fn
|
| 137 |
+
return fn
|
| 138 |
+
|
| 139 |
+
return decorate
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def get_aggregation(name):
|
| 143 |
+
try:
|
| 144 |
+
return AGGREGATION_REGISTRY[name]
|
| 145 |
+
except KeyError:
|
| 146 |
+
eval_logger.warning(
|
| 147 |
+
"{} not a registered aggregation metric!".format(name),
|
| 148 |
+
)
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
def get_metric_aggregation(name):
|
| 152 |
+
try:
|
| 153 |
+
return METRIC_AGGREGATION_REGISTRY[name]
|
| 154 |
+
except KeyError:
|
| 155 |
+
eval_logger.warning(
|
| 156 |
+
"{} metric is not assigned a default aggregation!".format(name),
|
| 157 |
+
)
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def is_higher_better(metric_name):
|
| 161 |
+
try:
|
| 162 |
+
return HIGHER_IS_BETTER_REGISTRY[metric_name]
|
| 163 |
+
except KeyError:
|
| 164 |
+
eval_logger.warning(f"higher_is_better not specified for metric '{metric_name}'!")
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
def register_filter(name):
|
| 168 |
+
def decorate(cls):
|
| 169 |
+
if name in FILTER_REGISTRY:
|
| 170 |
+
eval_logger.info(f"Registering filter `{name}` that is already in Registry {FILTER_REGISTRY}")
|
| 171 |
+
FILTER_REGISTRY[name] = cls
|
| 172 |
+
return cls
|
| 173 |
+
|
| 174 |
+
return decorate
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
def get_filter(filter_name: Union[str, Callable]) -> Callable:
|
| 178 |
+
try:
|
| 179 |
+
return FILTER_REGISTRY[filter_name]
|
| 180 |
+
except KeyError as e:
|
| 181 |
+
if callable(filter_name):
|
| 182 |
+
return filter_name
|
| 183 |
+
else:
|
| 184 |
+
eval_logger.warning(f"filter `{filter_name}` is not registered!")
|
| 185 |
+
raise e
|
LaViDa/eval/lmms_eval/api/samplers.py
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
class ContextSampler:
|
| 2 |
+
def __init__(self, docs, task, fewshot_indices=None, rnd=None) -> None:
|
| 3 |
+
self.rnd = rnd
|
| 4 |
+
assert self.rnd, "must pass rnd to FewShotSampler!"
|
| 5 |
+
|
| 6 |
+
self.task = task
|
| 7 |
+
self.config = task._config
|
| 8 |
+
|
| 9 |
+
self.target_delimiter = self.config.target_delimiter
|
| 10 |
+
self.fewshot_delimiter = self.config.fewshot_delimiter
|
| 11 |
+
|
| 12 |
+
self.doc_to_text = self.task.doc_to_text
|
| 13 |
+
self.doc_to_target = self.task.doc_to_target
|
| 14 |
+
self.doc_to_choice = self.task.doc_to_choice
|
| 15 |
+
|
| 16 |
+
self.docs = docs # HF dataset split, provided by task._fewshot_docs()
|
| 17 |
+
if fewshot_indices: # subset few-shot docs from
|
| 18 |
+
self.docs = self.docs.select(fewshot_indices)
|
| 19 |
+
|
| 20 |
+
def get_context(self, doc, num_fewshot):
|
| 21 |
+
# draw an extra fewshot sample if using same split as evaluating on
|
| 22 |
+
n_samples = num_fewshot + 1 if self.config.fewshot_split == self.config.test_split else num_fewshot
|
| 23 |
+
|
| 24 |
+
# draw `n_samples` docs from fewshot_docs
|
| 25 |
+
fewshotex = self.sample(n_samples)
|
| 26 |
+
|
| 27 |
+
# get rid of the doc that's the one we're evaluating, if it's in the fewshot
|
| 28 |
+
# TODO: should we just stop people from using fewshot from same split as evaluating?
|
| 29 |
+
selected_docs = [x for x in fewshotex if x != doc][:num_fewshot]
|
| 30 |
+
|
| 31 |
+
labeled_examples = (
|
| 32 |
+
self.fewshot_delimiter.join(
|
| 33 |
+
[
|
| 34 |
+
# TODO: is separating doc_to_text and doc_to_target by one space always desired?
|
| 35 |
+
(self.doc_to_text(doc) if (self.config.doc_to_choice is None or type(self.doc_to_text(doc)) is str) else self.doc_to_choice(doc)[self.doc_to_text(doc)])
|
| 36 |
+
+ self.target_delimiter
|
| 37 |
+
+ (
|
| 38 |
+
str(self.doc_to_target(doc)[0])
|
| 39 |
+
if type(self.doc_to_target(doc)) is list
|
| 40 |
+
else self.doc_to_target(doc)
|
| 41 |
+
if (self.config.doc_to_choice is None or type(self.doc_to_target(doc)) is str)
|
| 42 |
+
else str(self.doc_to_choice(doc)[self.doc_to_target(doc)])
|
| 43 |
+
)
|
| 44 |
+
for doc in selected_docs
|
| 45 |
+
]
|
| 46 |
+
)
|
| 47 |
+
+ self.fewshot_delimiter
|
| 48 |
+
)
|
| 49 |
+
|
| 50 |
+
return labeled_examples
|
| 51 |
+
|
| 52 |
+
def sample(self, n):
|
| 53 |
+
"""
|
| 54 |
+
Draw `n` samples from our fewshot docs. This method should be overridden by subclasses.
|
| 55 |
+
"""
|
| 56 |
+
|
| 57 |
+
return self.rnd.sample(self.docs, n)
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
class FirstNSampler(ContextSampler):
|
| 61 |
+
def sample(self, n) -> None:
|
| 62 |
+
"""
|
| 63 |
+
Draw the first `n` samples in order from the specified split.
|
| 64 |
+
Used for tasks with "canonical" ordered fewshot examples, such as MMLU and CMMLU.
|
| 65 |
+
"""
|
| 66 |
+
assert n <= len(self.docs), f"Error: number of fewshot samples requested exceeds the {len(self.docs)} that are available."
|
| 67 |
+
return self.docs[:n]
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
class BalancedSampler(ContextSampler):
|
| 71 |
+
def sample(self, n) -> None:
|
| 72 |
+
"""
|
| 73 |
+
TODO: this should return approximately class-balanced samples from our fewshot examples.
|
| 74 |
+
TODO: what order should they be in? maybe random?
|
| 75 |
+
"""
|
| 76 |
+
|
| 77 |
+
pass
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
class ManualSampler(ContextSampler):
|
| 81 |
+
def sample(self, n) -> None:
|
| 82 |
+
""" """
|
| 83 |
+
pass
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
SAMPLER_REGISTRY = {
|
| 87 |
+
"default": ContextSampler,
|
| 88 |
+
"first_n": FirstNSampler,
|
| 89 |
+
}
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def get_sampler(name):
|
| 93 |
+
try:
|
| 94 |
+
return SAMPLER_REGISTRY[name]
|
| 95 |
+
except KeyError:
|
| 96 |
+
raise ValueError(f"Attempted to use contextsampler '{name}', but no sampling strategy for this name found! Supported model names: {', '.join(SAMPLER_REGISTRY.keys())}")
|
LaViDa/eval/lmms_eval/api/task.py
ADDED
|
@@ -0,0 +1,1630 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import abc
|
| 2 |
+
import ast
|
| 3 |
+
import copy
|
| 4 |
+
import inspect
|
| 5 |
+
import itertools
|
| 6 |
+
import json
|
| 7 |
+
import os
|
| 8 |
+
import random
|
| 9 |
+
import re
|
| 10 |
+
import shutil
|
| 11 |
+
import subprocess
|
| 12 |
+
from collections.abc import Callable
|
| 13 |
+
from dataclasses import asdict, dataclass, field
|
| 14 |
+
from functools import partial
|
| 15 |
+
from glob import glob
|
| 16 |
+
from typing import (
|
| 17 |
+
Any,
|
| 18 |
+
Dict,
|
| 19 |
+
Iterable,
|
| 20 |
+
Iterator,
|
| 21 |
+
List,
|
| 22 |
+
Literal,
|
| 23 |
+
Mapping,
|
| 24 |
+
Optional,
|
| 25 |
+
Tuple,
|
| 26 |
+
Union,
|
| 27 |
+
)
|
| 28 |
+
|
| 29 |
+
import datasets
|
| 30 |
+
import numpy as np
|
| 31 |
+
from accelerate import Accelerator
|
| 32 |
+
from datasets import Audio, DownloadConfig, Image, Sequence
|
| 33 |
+
from huggingface_hub import snapshot_download
|
| 34 |
+
from loguru import logger as eval_logger
|
| 35 |
+
from PIL import ImageFile
|
| 36 |
+
from tenacity import retry, stop_after_attempt, stop_after_delay, wait_fixed
|
| 37 |
+
from tqdm import tqdm
|
| 38 |
+
|
| 39 |
+
from lmms_eval import utils
|
| 40 |
+
from lmms_eval.api import samplers
|
| 41 |
+
from lmms_eval.api.instance import Instance
|
| 42 |
+
from lmms_eval.api.registry import (
|
| 43 |
+
AGGREGATION_REGISTRY,
|
| 44 |
+
DEFAULT_METRIC_REGISTRY,
|
| 45 |
+
METRIC_REGISTRY,
|
| 46 |
+
OUTPUT_TYPE_REGISTRY,
|
| 47 |
+
get_aggregation,
|
| 48 |
+
get_metric,
|
| 49 |
+
get_metric_aggregation,
|
| 50 |
+
is_higher_better,
|
| 51 |
+
)
|
| 52 |
+
from lmms_eval.caching.cache import load_from_cache, save_to_cache
|
| 53 |
+
from lmms_eval.filters import build_filter_ensemble
|
| 54 |
+
|
| 55 |
+
# HuggingfaceM4/NoCaps contains truncated image in test split
|
| 56 |
+
# Include this inside code block to avoid error
|
| 57 |
+
ImageFile.LOAD_TRUNCATED_IMAGES = True
|
| 58 |
+
|
| 59 |
+
ALL_OUTPUT_TYPES = [
|
| 60 |
+
"loglikelihood",
|
| 61 |
+
"multiple_choice",
|
| 62 |
+
"generate_until",
|
| 63 |
+
"generate_until_multi_round",
|
| 64 |
+
]
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
@dataclass
|
| 68 |
+
class TaskConfig(dict):
|
| 69 |
+
# task naming/registry
|
| 70 |
+
task: str = None
|
| 71 |
+
task_alias: str = None
|
| 72 |
+
tag: str = None
|
| 73 |
+
group: Union[str, list] = None
|
| 74 |
+
group_alias: Union[str, list] = None
|
| 75 |
+
# HF dataset options.
|
| 76 |
+
# which dataset to use,
|
| 77 |
+
# and what splits for what purpose
|
| 78 |
+
dataset_path: str = None
|
| 79 |
+
dataset_name: str = None
|
| 80 |
+
dataset_kwargs: dict = None
|
| 81 |
+
training_split: str = None
|
| 82 |
+
validation_split: str = None
|
| 83 |
+
test_split: str = None
|
| 84 |
+
fewshot_split: str = None # TODO: assert that this not None if num_fewshot > 0. (?) assert if this is same split as one evaling (?)
|
| 85 |
+
full_docs: bool = False
|
| 86 |
+
# formatting / prompting options.
|
| 87 |
+
# see docs/advanced_task_guide.md for more info
|
| 88 |
+
process_results_use_image: bool = False
|
| 89 |
+
process_docs: Callable = None
|
| 90 |
+
doc_to_visual: Union[Callable, str] = None
|
| 91 |
+
doc_to_text: Union[Callable, str] = None
|
| 92 |
+
doc_to_target: Union[Callable, str] = None
|
| 93 |
+
doc_to_choice: Union[Callable, str, dict, list] = None
|
| 94 |
+
process_results: Union[Callable, str] = None
|
| 95 |
+
use_prompt: str = None
|
| 96 |
+
description: str = ""
|
| 97 |
+
target_delimiter: str = " "
|
| 98 |
+
fewshot_delimiter: str = "\n\n"
|
| 99 |
+
fewshot_config: dict = None
|
| 100 |
+
# runtime configuration options
|
| 101 |
+
num_fewshot: int = None
|
| 102 |
+
# scoring options
|
| 103 |
+
metric_list: list = None
|
| 104 |
+
output_type: str = "generate_until"
|
| 105 |
+
generation_kwargs: dict = None
|
| 106 |
+
repeats: int = 1
|
| 107 |
+
filter_list: Union[str, list] = None
|
| 108 |
+
should_decontaminate: bool = False
|
| 109 |
+
doc_to_decontamination_query: str = None
|
| 110 |
+
|
| 111 |
+
metadata: Union[str, list] = None # by default, not used in the code. allows for users to pass arbitrary info to tasks
|
| 112 |
+
|
| 113 |
+
lmms_eval_specific_kwargs: dict = None
|
| 114 |
+
model_specific_generation_kwargs: dict = None
|
| 115 |
+
model_specific_target_kwargs: dict = None
|
| 116 |
+
|
| 117 |
+
def __post_init__(self) -> None:
|
| 118 |
+
if self.dataset_path and os.path.exists(os.path.dirname(self.dataset_path)):
|
| 119 |
+
import inspect
|
| 120 |
+
from importlib import import_module
|
| 121 |
+
|
| 122 |
+
# self.dataset_path = inspect.getfile(import_module(self.dataset_path))
|
| 123 |
+
|
| 124 |
+
if self.group is not None:
|
| 125 |
+
eval_logger.warning(
|
| 126 |
+
"A task YAML file was found to contain a `group` key. Groups which provide aggregate scores over several subtasks now require a separate config file--if not aggregating, you may want to use the `tag` config option instead within your config. Setting `group` within a TaskConfig will be deprecated in v0.4.4. Please see https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md for more information."
|
| 127 |
+
)
|
| 128 |
+
|
| 129 |
+
if self.tag is None:
|
| 130 |
+
self.tag = self.group
|
| 131 |
+
else:
|
| 132 |
+
raise ValueError("Got both a `group` and `tag` entry within a TaskConfig. Please use one or the other--`group` values will be deprecated in v0.4.4.")
|
| 133 |
+
|
| 134 |
+
if self.generation_kwargs is not None:
|
| 135 |
+
if "generate_until" not in self.output_type:
|
| 136 |
+
eval_logger.warning(f"[{self.task}] passed `generation_kwargs`, but not using `output_type: generate_until`!")
|
| 137 |
+
assert "generate_until" not in self.output_type
|
| 138 |
+
|
| 139 |
+
if "temperature" in self.generation_kwargs:
|
| 140 |
+
self.generation_kwargs["temperature"] = float(self.generation_kwargs["temperature"])
|
| 141 |
+
|
| 142 |
+
if "until" not in self.generation_kwargs:
|
| 143 |
+
self.generation_kwargs["until"] = [self.fewshot_delimiter]
|
| 144 |
+
else:
|
| 145 |
+
if "generate_until" in self.output_type:
|
| 146 |
+
# ensure that we greedily generate in absence of explicit arguments otherwise
|
| 147 |
+
self.generation_kwargs = {
|
| 148 |
+
"until": None if self.fewshot_delimiter is None else [self.fewshot_delimiter],
|
| 149 |
+
"do_sample": False,
|
| 150 |
+
}
|
| 151 |
+
|
| 152 |
+
# TODO: how to make TaskConfigs be de- and re-serializable, even when using the !function constructor?
|
| 153 |
+
|
| 154 |
+
def __getitem__(self, item):
|
| 155 |
+
return getattr(self, item)
|
| 156 |
+
|
| 157 |
+
def __setitem__(self, item, value):
|
| 158 |
+
return setattr(self, item, value)
|
| 159 |
+
|
| 160 |
+
def to_dict(self):
|
| 161 |
+
"""dumps the current config as a dictionary object, as a printable format.
|
| 162 |
+
null fields will not be printed.
|
| 163 |
+
Used for dumping results alongside full task configuration
|
| 164 |
+
|
| 165 |
+
:return: dict
|
| 166 |
+
A printable dictionary version of the TaskConfig object.
|
| 167 |
+
|
| 168 |
+
# TODO: should any default value in the TaskConfig not be printed?
|
| 169 |
+
"""
|
| 170 |
+
cfg_dict = asdict(self)
|
| 171 |
+
# remove values that are `None`
|
| 172 |
+
for k, v in list(cfg_dict.items()):
|
| 173 |
+
if v is None:
|
| 174 |
+
cfg_dict.pop(k)
|
| 175 |
+
elif isinstance(v, Callable):
|
| 176 |
+
# TODO: this should handle Promptsource template objects as a separate case?
|
| 177 |
+
cfg_dict[k] = str(v)
|
| 178 |
+
return cfg_dict
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
class Task(abc.ABC):
|
| 182 |
+
"""A task represents an entire benchmark including its dataset, problems,
|
| 183 |
+
answers, and evaluation methods. See BoolQ for a simple example implementation
|
| 184 |
+
|
| 185 |
+
A `doc` can be any python object which represents one instance of evaluation.
|
| 186 |
+
This is usually a dictionary e.g.
|
| 187 |
+
{"question": ..., "answer": ...} or
|
| 188 |
+
{"question": ..., question, answer)
|
| 189 |
+
"""
|
| 190 |
+
|
| 191 |
+
VERSION = None
|
| 192 |
+
|
| 193 |
+
# The name of the `Task` benchmark as denoted in the HuggingFace datasets Hub
|
| 194 |
+
# or a path to a custom `datasets` loading script.
|
| 195 |
+
DATASET_PATH: str = None
|
| 196 |
+
|
| 197 |
+
# The name of a subset within `DATASET_PATH`.
|
| 198 |
+
DATASET_NAME: str = None
|
| 199 |
+
|
| 200 |
+
OUTPUT_TYPE: str = None
|
| 201 |
+
|
| 202 |
+
def __init__(
|
| 203 |
+
self,
|
| 204 |
+
data_dir=None,
|
| 205 |
+
cache_dir=None,
|
| 206 |
+
download_mode=None,
|
| 207 |
+
config=None,
|
| 208 |
+
) -> None:
|
| 209 |
+
"""
|
| 210 |
+
:param data_dir: str
|
| 211 |
+
Stores the path to a local folder containing the `Task`'s data files.
|
| 212 |
+
Use this to specify the path to manually downloaded data (usually when
|
| 213 |
+
the dataset is not publicly accessible).
|
| 214 |
+
:param cache_dir: str
|
| 215 |
+
The directory to read/write the `Task` dataset. This follows the
|
| 216 |
+
HuggingFace `datasets` API with the default cache directory located at:
|
| 217 |
+
`~/.cache/huggingface/datasets`
|
| 218 |
+
NOTE: You can change the cache location globally for a given process
|
| 219 |
+
to another directory:
|
| 220 |
+
`export HF_DATASETS_CACHE="/path/to/another/directory"`
|
| 221 |
+
:param download_mode: datasets.DownloadMode
|
| 222 |
+
How to treat pre-existing `Task` downloads and data.
|
| 223 |
+
- `datasets.DownloadMode.REUSE_DATASET_IF_EXISTS`
|
| 224 |
+
Reuse download and reuse dataset.
|
| 225 |
+
- `datasets.DownloadMode.REUSE_CACHE_IF_EXISTS`
|
| 226 |
+
Reuse download with fresh dataset.
|
| 227 |
+
- `datasets.DownloadMode.FORCE_REDOWNLOAD`
|
| 228 |
+
Fresh download and fresh dataset.
|
| 229 |
+
"""
|
| 230 |
+
self.download(data_dir, cache_dir, download_mode)
|
| 231 |
+
self._training_docs = None
|
| 232 |
+
self._fewshot_docs = None
|
| 233 |
+
self._instances = None
|
| 234 |
+
|
| 235 |
+
self._config = TaskConfig({**config}) if config else TaskConfig()
|
| 236 |
+
|
| 237 |
+
self._filters = [build_filter_ensemble("none", [["take_first", None]])]
|
| 238 |
+
|
| 239 |
+
def download(self, data_dir=None, cache_dir=None, download_mode=None) -> None:
|
| 240 |
+
"""Downloads and returns the task dataset.
|
| 241 |
+
Override this method to download the dataset from a custom API.
|
| 242 |
+
|
| 243 |
+
:param data_dir: str
|
| 244 |
+
Stores the path to a local folder containing the `Task`'s data files.
|
| 245 |
+
Use this to specify the path to manually downloaded data (usually when
|
| 246 |
+
the dataset is not publicly accessible).
|
| 247 |
+
:param cache_dir: str
|
| 248 |
+
The directory to read/write the `Task` dataset. This follows the
|
| 249 |
+
HuggingFace `datasets` API with the default cache directory located at:
|
| 250 |
+
`~/.cache/huggingface/datasets`
|
| 251 |
+
NOTE: You can change the cache location globally for a given process
|
| 252 |
+
by setting the shell environment variable, `HF_DATASETS_CACHE`,
|
| 253 |
+
to another directory:
|
| 254 |
+
`export HF_DATASETS_CACHE="/path/to/another/directory"`
|
| 255 |
+
:param download_mode: datasets.DownloadMode
|
| 256 |
+
How to treat pre-existing `Task` downloads and data.
|
| 257 |
+
- `datasets.DownloadMode.REUSE_DATASET_IF_EXISTS`
|
| 258 |
+
Reuse download and reuse dataset.
|
| 259 |
+
- `datasets.DownloadMode.REUSE_CACHE_IF_EXISTS`
|
| 260 |
+
Reuse download with fresh dataset.
|
| 261 |
+
- `datasets.DownloadMode.FORCE_REDOWNLOAD`
|
| 262 |
+
Fresh download and fresh dataset.
|
| 263 |
+
"""
|
| 264 |
+
self.dataset = datasets.load_dataset(
|
| 265 |
+
path=self.DATASET_PATH,
|
| 266 |
+
name=self.DATASET_NAME,
|
| 267 |
+
data_dir=data_dir,
|
| 268 |
+
cache_dir=cache_dir,
|
| 269 |
+
download_mode=download_mode,
|
| 270 |
+
)
|
| 271 |
+
self.dataset_no_image = datasets.load_dataset(
|
| 272 |
+
path=self.DATASET_PATH,
|
| 273 |
+
name=self.DATASET_NAME,
|
| 274 |
+
data_dir=data_dir,
|
| 275 |
+
cache_dir=cache_dir,
|
| 276 |
+
download_mode=download_mode,
|
| 277 |
+
)
|
| 278 |
+
for doc_name in self.dataset_no_image:
|
| 279 |
+
remove_cols = []
|
| 280 |
+
features = self.dataset_no_image[doc_name].features
|
| 281 |
+
# If it is an Image instance or a Sequence of Image instance. Remove it
|
| 282 |
+
for feature in features:
|
| 283 |
+
if isinstance(features[feature], Image):
|
| 284 |
+
remove_cols.append(feature)
|
| 285 |
+
elif isinstance(features[feature], Sequence) and isinstance(features[feature].feature, Image):
|
| 286 |
+
remove_cols.append(feature)
|
| 287 |
+
for remove_col in remove_cols:
|
| 288 |
+
self.dataset_no_image[doc_name] = self.dataset_no_image[doc_name].remove_columns(remove_col)
|
| 289 |
+
|
| 290 |
+
@property
|
| 291 |
+
def config(self):
|
| 292 |
+
"""Returns the TaskConfig associated with this class."""
|
| 293 |
+
return self._config
|
| 294 |
+
|
| 295 |
+
@abc.abstractmethod
|
| 296 |
+
def has_training_docs(self):
|
| 297 |
+
"""Whether the task has a training set"""
|
| 298 |
+
pass
|
| 299 |
+
|
| 300 |
+
@abc.abstractmethod
|
| 301 |
+
def has_validation_docs(self):
|
| 302 |
+
"""Whether the task has a validation set"""
|
| 303 |
+
pass
|
| 304 |
+
|
| 305 |
+
@abc.abstractmethod
|
| 306 |
+
def has_test_docs(self):
|
| 307 |
+
"""Whether the task has a test set"""
|
| 308 |
+
pass
|
| 309 |
+
|
| 310 |
+
def training_docs(self):
|
| 311 |
+
"""
|
| 312 |
+
:return: Iterable[obj]
|
| 313 |
+
A iterable of any object, that doc_to_text can handle
|
| 314 |
+
"""
|
| 315 |
+
return []
|
| 316 |
+
|
| 317 |
+
def validation_docs(self):
|
| 318 |
+
"""
|
| 319 |
+
:return: Iterable[obj]
|
| 320 |
+
A iterable of any object, that doc_to_text can handle
|
| 321 |
+
"""
|
| 322 |
+
return []
|
| 323 |
+
|
| 324 |
+
def test_docs(self):
|
| 325 |
+
"""
|
| 326 |
+
:return: Iterable[obj]
|
| 327 |
+
A iterable of any object, that doc_to_text can handle
|
| 328 |
+
"""
|
| 329 |
+
return []
|
| 330 |
+
|
| 331 |
+
def fewshot_docs(self):
|
| 332 |
+
"""
|
| 333 |
+
:return: Iterable[obj]
|
| 334 |
+
A iterable of any object, that doc_to_text can handle
|
| 335 |
+
"""
|
| 336 |
+
if self.has_training_docs():
|
| 337 |
+
return self.training_docs()
|
| 338 |
+
elif self.has_validation_docs():
|
| 339 |
+
return self.validation_docs()
|
| 340 |
+
else:
|
| 341 |
+
if self.config.num_fewshot is not None:
|
| 342 |
+
eval_logger.warning("has_training_docs and has_validation_docs are False" ", using test_docs as fewshot_docs but this is not recommended.")
|
| 343 |
+
return self.test_docs()
|
| 344 |
+
|
| 345 |
+
def _process_doc(self, doc):
|
| 346 |
+
"""
|
| 347 |
+
Override this to process (detokenize, strip, replace, etc.) individual
|
| 348 |
+
documents. This can be used in a map over documents of a data split.
|
| 349 |
+
E.g. `map(self._process_doc, self.dataset["validation"])`
|
| 350 |
+
|
| 351 |
+
:return: dict
|
| 352 |
+
The processed version of the specified `doc`.
|
| 353 |
+
"""
|
| 354 |
+
return doc
|
| 355 |
+
|
| 356 |
+
@property
|
| 357 |
+
def instances(self):
|
| 358 |
+
"""After calling `task.build_all_requests()`, tasks
|
| 359 |
+
maintain a list of the dataset instances which will be evaluated.
|
| 360 |
+
"""
|
| 361 |
+
return self._instances
|
| 362 |
+
|
| 363 |
+
def fewshot_examples(self, k, rnd):
|
| 364 |
+
if self._training_docs is None:
|
| 365 |
+
self._training_docs = list(self.training_docs())
|
| 366 |
+
|
| 367 |
+
return rnd.sample(self._training_docs, k)
|
| 368 |
+
|
| 369 |
+
def doc_to_decontamination_query(self, doc) -> None:
|
| 370 |
+
print("Override doc_to_decontamination_query with document specific decontamination query.")
|
| 371 |
+
assert False
|
| 372 |
+
|
| 373 |
+
@abc.abstractmethod
|
| 374 |
+
def doc_to_text(self, doc):
|
| 375 |
+
pass
|
| 376 |
+
|
| 377 |
+
@abc.abstractmethod
|
| 378 |
+
def doc_to_target(self, doc):
|
| 379 |
+
pass
|
| 380 |
+
|
| 381 |
+
# @profile
|
| 382 |
+
def build_all_requests(
|
| 383 |
+
self,
|
| 384 |
+
*,
|
| 385 |
+
limit: Union[int, None] = None,
|
| 386 |
+
rank: int = 0,
|
| 387 |
+
world_size: int = 1,
|
| 388 |
+
cache_requests: bool = False,
|
| 389 |
+
rewrite_requests_cache: bool = False,
|
| 390 |
+
system_instruction: Optional[str] = None,
|
| 391 |
+
apply_chat_template: bool = False,
|
| 392 |
+
fewshot_as_multiturn: bool = False,
|
| 393 |
+
chat_template: Optional[Callable] = None,
|
| 394 |
+
tokenizer_name: str = "",
|
| 395 |
+
) -> None:
|
| 396 |
+
"""Build a set of Instances for a task, and store them in task.instances"""
|
| 397 |
+
if self.has_test_docs():
|
| 398 |
+
docs = self.test_docs()
|
| 399 |
+
split = self.config.test_split
|
| 400 |
+
elif self.has_validation_docs():
|
| 401 |
+
docs = self.validation_docs()
|
| 402 |
+
split = self.config.validation_split
|
| 403 |
+
else:
|
| 404 |
+
assert False, f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!"
|
| 405 |
+
|
| 406 |
+
# used with caching
|
| 407 |
+
og_limit = limit
|
| 408 |
+
|
| 409 |
+
cache_key = f"requests-{self._config.task}-{self.config.num_fewshot}shot-rank{rank}-world_size{world_size}"
|
| 410 |
+
cache_key += "-chat_template" if apply_chat_template else ""
|
| 411 |
+
cache_key += "-fewshot_as_multiturn" if fewshot_as_multiturn else ""
|
| 412 |
+
cache_key += f"-system_prompt_hash{utils.hash_string(system_instruction)}" if system_instruction is not None else ""
|
| 413 |
+
cache_key += f"-tokenizer{tokenizer_name}"
|
| 414 |
+
|
| 415 |
+
cached_instances = load_from_cache(file_name=cache_key)
|
| 416 |
+
|
| 417 |
+
if cache_requests and cached_instances and not rewrite_requests_cache:
|
| 418 |
+
cached_instances = cached_instances[:limit]
|
| 419 |
+
|
| 420 |
+
flattened_instances = [instance for instance_group in cached_instances for instance in instance_group]
|
| 421 |
+
|
| 422 |
+
self._instances = flattened_instances
|
| 423 |
+
return
|
| 424 |
+
|
| 425 |
+
eval_logger.info(f"Building contexts for {self.config.task} on rank {rank}...")
|
| 426 |
+
|
| 427 |
+
instances = []
|
| 428 |
+
|
| 429 |
+
# process all documents when caching is specified for simplicity
|
| 430 |
+
if cache_requests and (not cached_instances or rewrite_requests_cache) and limit is not None:
|
| 431 |
+
limit = None
|
| 432 |
+
|
| 433 |
+
doc_id_docs = utils.create_iterator(enumerate(self.eval_docs_no_media), rank=rank, limit=int(limit) if limit else None, world_size=world_size)
|
| 434 |
+
doc_iterator_for_counting = itertools.islice(range(len(self.test_docs())), rank, limit, world_size) if self.has_test_docs() else itertools.islice(range(len(self.validation_docs())), rank, limit, world_size)
|
| 435 |
+
|
| 436 |
+
num_docs = sum(1 for _ in doc_iterator_for_counting)
|
| 437 |
+
|
| 438 |
+
for doc_id, doc in tqdm(
|
| 439 |
+
doc_id_docs,
|
| 440 |
+
total=num_docs,
|
| 441 |
+
):
|
| 442 |
+
# sample fewshot context #TODO: need to offset doc_id by rank now!
|
| 443 |
+
fewshot_ctx = self.fewshot_context(
|
| 444 |
+
doc,
|
| 445 |
+
0 if self.config.num_fewshot is None else self.config.num_fewshot,
|
| 446 |
+
system_instruction,
|
| 447 |
+
apply_chat_template,
|
| 448 |
+
fewshot_as_multiturn,
|
| 449 |
+
chat_template,
|
| 450 |
+
)
|
| 451 |
+
|
| 452 |
+
# TODO: we should override self.config.repeats if doing greedy gen so users don't waste time+compute
|
| 453 |
+
per_task_metadata = {"task": self.config["task"], "doc_id": doc_id, "repeats": self.config.repeats, "split": split}
|
| 454 |
+
if self.config.metadata and type(self.config.metadata) == dict: # TODO: temporary fix for metadata loading, ignore the list of dict type.
|
| 455 |
+
per_task_metadata.update(self.config.metadata)
|
| 456 |
+
|
| 457 |
+
inst = self.construct_requests(doc_id=doc_id, ctx=fewshot_ctx, metadata=per_task_metadata)
|
| 458 |
+
|
| 459 |
+
if not isinstance(inst, list):
|
| 460 |
+
inst = [inst]
|
| 461 |
+
|
| 462 |
+
instances.append(inst)
|
| 463 |
+
|
| 464 |
+
# now flatten, this is to allow slicing to work with pickles
|
| 465 |
+
|
| 466 |
+
sliced_instances = instances[:og_limit]
|
| 467 |
+
|
| 468 |
+
flattened_instances = [instance for instance_group in sliced_instances for instance in instance_group]
|
| 469 |
+
|
| 470 |
+
self._instances = flattened_instances
|
| 471 |
+
|
| 472 |
+
if len(self._instances) == 0:
|
| 473 |
+
raise ValueError("task.build_requests() did not find any docs!")
|
| 474 |
+
|
| 475 |
+
if cache_requests and (not cached_instances or rewrite_requests_cache):
|
| 476 |
+
save_to_cache(file_name=cache_key, obj=instances)
|
| 477 |
+
|
| 478 |
+
# FIXME: Bo - We need to check if the doc_to_visual if it's exists and restore it. If we use cache, the doc_to_visual will be None since it's not serializable
|
| 479 |
+
for instance in self._instances:
|
| 480 |
+
if instance.arguments[2] is None:
|
| 481 |
+
arguments = (instance.arguments[0], instance.arguments[1], self.doc_to_visual, *instance.arguments[3:])
|
| 482 |
+
else:
|
| 483 |
+
arguments = instance.arguments
|
| 484 |
+
|
| 485 |
+
instance.arguments = arguments
|
| 486 |
+
|
| 487 |
+
@abc.abstractmethod
|
| 488 |
+
def construct_requests(self, doc_id, ctx, **kwargs):
|
| 489 |
+
"""Uses RequestFactory to construct Requests and returns an iterable of
|
| 490 |
+
Requests which will be sent to the LMM.
|
| 491 |
+
|
| 492 |
+
:param doc_id: int
|
| 493 |
+
The index of a document within `self.test_docs()` or `self.validation_docs()`,
|
| 494 |
+
whichever is the main split used.
|
| 495 |
+
:param ctx: str
|
| 496 |
+
The context string, generated by fewshot_context. This includes the natural
|
| 497 |
+
language description, as well as the few shot examples, and the question
|
| 498 |
+
part of the document for `doc`.
|
| 499 |
+
:param repeats: int
|
| 500 |
+
TODO: update this docstring
|
| 501 |
+
The number of times each instance in a dataset is inferred on. Defaults to 1,
|
| 502 |
+
can be increased for techniques like majority voting.
|
| 503 |
+
"""
|
| 504 |
+
pass
|
| 505 |
+
|
| 506 |
+
@abc.abstractmethod
|
| 507 |
+
def process_results(self, doc, results):
|
| 508 |
+
"""Take a single document and the LMM results and evaluates, returning a
|
| 509 |
+
dict where keys are the names of submetrics and values are the values of
|
| 510 |
+
the metric for that one document
|
| 511 |
+
|
| 512 |
+
:param doc:
|
| 513 |
+
The document as returned from training_docs, validation_docs, or test_docs.
|
| 514 |
+
:param results:
|
| 515 |
+
The results of the requests created in construct_requests.
|
| 516 |
+
"""
|
| 517 |
+
pass
|
| 518 |
+
|
| 519 |
+
@abc.abstractmethod
|
| 520 |
+
def aggregation(self):
|
| 521 |
+
"""
|
| 522 |
+
:returns: {str: [metric_score] -> float}
|
| 523 |
+
A dictionary where keys are the names of submetrics and values are
|
| 524 |
+
functions that aggregate a list of metric scores
|
| 525 |
+
"""
|
| 526 |
+
pass
|
| 527 |
+
|
| 528 |
+
@abc.abstractmethod
|
| 529 |
+
def higher_is_better(self):
|
| 530 |
+
"""
|
| 531 |
+
:returns: {str: bool}
|
| 532 |
+
A dictionary where keys are the names of submetrics and values are
|
| 533 |
+
whether a higher value of the submetric is better
|
| 534 |
+
"""
|
| 535 |
+
pass
|
| 536 |
+
|
| 537 |
+
@classmethod
|
| 538 |
+
def count_bytes(cls, doc):
|
| 539 |
+
"""Used for byte-level perplexity metrics in rolling loglikelihood"""
|
| 540 |
+
return len(doc.encode("utf-8"))
|
| 541 |
+
|
| 542 |
+
@utils.positional_deprecated
|
| 543 |
+
def fewshot_context(
|
| 544 |
+
self,
|
| 545 |
+
doc_id,
|
| 546 |
+
num_fewshot,
|
| 547 |
+
split,
|
| 548 |
+
rnd=random.Random(1234),
|
| 549 |
+
description=None,
|
| 550 |
+
):
|
| 551 |
+
"""Returns a fewshot context string that is made up of a prepended description
|
| 552 |
+
(if provided), the `num_fewshot` number of examples, and an appended prompt example.
|
| 553 |
+
|
| 554 |
+
:param doc_id: int
|
| 555 |
+
The document id as returned from training_docs, validation_docs, or test_docs.
|
| 556 |
+
:param num_fewshot: int
|
| 557 |
+
The number of fewshot examples to provide in the returned context string.
|
| 558 |
+
:param split: str
|
| 559 |
+
The split of the document to retrieve from the dataset
|
| 560 |
+
:param rnd: random.Random
|
| 561 |
+
The pseudo-random number generator used to randomly sample examples.
|
| 562 |
+
WARNING: This is currently a required arg although it's optionalized with a default `None`.
|
| 563 |
+
:param description: str
|
| 564 |
+
The task's description that will be prepended to the fewshot examples.
|
| 565 |
+
:returns: str
|
| 566 |
+
The fewshot context.
|
| 567 |
+
"""
|
| 568 |
+
assert rnd is not None, "A `random.Random` generator argument must be provided to `rnd`"
|
| 569 |
+
|
| 570 |
+
description = description if description else ""
|
| 571 |
+
doc = self.dataset_no_image[split][doc_id]
|
| 572 |
+
|
| 573 |
+
if num_fewshot == 0:
|
| 574 |
+
labeled_examples = ""
|
| 575 |
+
else:
|
| 576 |
+
# for sets with no training docs, draw from other set *but ensure no overlap with current doc*
|
| 577 |
+
if self.has_training_docs():
|
| 578 |
+
fewshotex = self.fewshot_examples(k=num_fewshot, rnd=rnd)
|
| 579 |
+
else:
|
| 580 |
+
if self._fewshot_docs is None:
|
| 581 |
+
self._fewshot_docs = list(self.validation_docs() if self.has_validation_docs() else self.test_docs())
|
| 582 |
+
|
| 583 |
+
fewshotex = rnd.sample(self._fewshot_docs, num_fewshot + 1)
|
| 584 |
+
|
| 585 |
+
# get rid of the doc that's the one we're evaluating, if it's in the fewshot
|
| 586 |
+
fewshotex = [x for x in fewshotex if x != doc][:num_fewshot]
|
| 587 |
+
|
| 588 |
+
labeled_examples = "\n\n".join([self.doc_to_text(doc) + self.doc_to_target(doc) for doc in fewshotex]) + "\n\n"
|
| 589 |
+
|
| 590 |
+
example = self.doc_to_text(doc)
|
| 591 |
+
return description + labeled_examples + example
|
| 592 |
+
|
| 593 |
+
def apply_filters(self) -> Optional[List[Instance]]:
|
| 594 |
+
"""Iterates over FilterEnsembles and applies them to instances"""
|
| 595 |
+
if hasattr(self, "_filters"):
|
| 596 |
+
for f in self._filters:
|
| 597 |
+
f.apply(self._instances, None)
|
| 598 |
+
else:
|
| 599 |
+
eval_logger.warning("No filter defined, passing through instances")
|
| 600 |
+
return self._instances
|
| 601 |
+
|
| 602 |
+
def dump_config(self) -> dict:
|
| 603 |
+
"""Returns a dictionary representing the task's config.
|
| 604 |
+
|
| 605 |
+
:returns: str
|
| 606 |
+
The fewshot context.
|
| 607 |
+
"""
|
| 608 |
+
# TODO: this should only return the overrides applied to a non-YAML task's configuration.
|
| 609 |
+
# (num_fewshot)
|
| 610 |
+
return self.config.to_dict()
|
| 611 |
+
|
| 612 |
+
def set_config(self, key: str, value: Any, update: bool = False) -> None:
|
| 613 |
+
"""Set or update the configuration for a given key."""
|
| 614 |
+
if key is None:
|
| 615 |
+
raise ValueError("Key must be provided.")
|
| 616 |
+
|
| 617 |
+
if update:
|
| 618 |
+
current_value = getattr(self._config, key, {})
|
| 619 |
+
if not isinstance(current_value, dict):
|
| 620 |
+
raise TypeError(f"Expected a dict for key '{key}', got {type(current_value).__name__} instead.")
|
| 621 |
+
current_value.update(value)
|
| 622 |
+
else:
|
| 623 |
+
setattr(self._config, key, value)
|
| 624 |
+
|
| 625 |
+
def override_metric(self, metric_name: str) -> None:
|
| 626 |
+
"""
|
| 627 |
+
Override the default metrics used for evaluation with custom metrics.
|
| 628 |
+
|
| 629 |
+
Parameters:
|
| 630 |
+
- metric_name (str): The name of the custom metric to override. Should be registered in api.metrics.
|
| 631 |
+
"""
|
| 632 |
+
(
|
| 633 |
+
self._metric_fn_list,
|
| 634 |
+
self._aggregation_list,
|
| 635 |
+
self._metric_fn_kwargs,
|
| 636 |
+
self._higher_is_better,
|
| 637 |
+
) = ({}, {}, {}, {})
|
| 638 |
+
self._metric_fn_list[metric_name] = get_metric(metric_name)
|
| 639 |
+
self._aggregation_list[metric_name] = get_metric_aggregation(metric_name)
|
| 640 |
+
self._higher_is_better[metric_name] = is_higher_better(metric_name)
|
| 641 |
+
self._metric_fn_kwargs[metric_name] = {}
|
| 642 |
+
if not isinstance(self, ConfigurableTask):
|
| 643 |
+
self.process_results = lambda x, y: {metric_name: get_metric(metric_name)}
|
| 644 |
+
self.aggregation = lambda: {metric_name: get_metric_aggregation(metric_name)}
|
| 645 |
+
setattr(self._config, "metric_list", [{"metric": metric_name}])
|
| 646 |
+
setattr(self._config, "process_results", None)
|
| 647 |
+
|
| 648 |
+
def set_fewshot_seed(self, seed: Optional[int] = None) -> None:
|
| 649 |
+
self.fewshot_rnd = random.Random(seed)
|
| 650 |
+
if hasattr(self, "sampler"):
|
| 651 |
+
self.sampler.rnd = self.fewshot_rnd
|
| 652 |
+
|
| 653 |
+
@property
|
| 654 |
+
def eval_docs(self) -> Union[datasets.Dataset, List[dict]]:
|
| 655 |
+
if self.has_test_docs():
|
| 656 |
+
return self.test_docs()
|
| 657 |
+
elif self.has_validation_docs():
|
| 658 |
+
return self.validation_docs()
|
| 659 |
+
else:
|
| 660 |
+
raise ValueError(f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!")
|
| 661 |
+
|
| 662 |
+
def doc_iterator(self, *, rank: int = 0, limit: Union[int, None] = None, world_size: int = 1) -> Iterator[Tuple[int, Any]]:
|
| 663 |
+
limit = int(limit) if limit else None
|
| 664 |
+
doc_iterator = utils.create_iterator(
|
| 665 |
+
enumerate(self.eval_docs),
|
| 666 |
+
rank=int(rank),
|
| 667 |
+
limit=limit,
|
| 668 |
+
world_size=int(world_size),
|
| 669 |
+
)
|
| 670 |
+
return doc_iterator
|
| 671 |
+
|
| 672 |
+
|
| 673 |
+
class ConfigurableTask(Task):
|
| 674 |
+
VERSION = "Yaml"
|
| 675 |
+
OUTPUT_TYPE = None
|
| 676 |
+
CONFIG = None
|
| 677 |
+
|
| 678 |
+
def __init__(
|
| 679 |
+
self,
|
| 680 |
+
data_dir=None,
|
| 681 |
+
cache_dir=None,
|
| 682 |
+
download_mode=None,
|
| 683 |
+
config: Optional[dict] = None,
|
| 684 |
+
model_name: Optional[str] = None,
|
| 685 |
+
) -> None: # TODO no super() call here
|
| 686 |
+
# Get pre-configured attributes
|
| 687 |
+
self._config = self.CONFIG
|
| 688 |
+
|
| 689 |
+
# Use new configurations if there was no preconfiguration
|
| 690 |
+
if self.config is None:
|
| 691 |
+
self._config = TaskConfig(**config)
|
| 692 |
+
# Overwrite configs
|
| 693 |
+
else:
|
| 694 |
+
if config is not None:
|
| 695 |
+
self._config.__dict__.update(config)
|
| 696 |
+
|
| 697 |
+
if self.config is None:
|
| 698 |
+
raise ValueError("Must pass a config to ConfigurableTask, either in cls.CONFIG or `config` kwarg")
|
| 699 |
+
|
| 700 |
+
if isinstance(self.config.metadata, dict):
|
| 701 |
+
if "version" in self.config.metadata:
|
| 702 |
+
self.VERSION = self.config.metadata["version"]
|
| 703 |
+
|
| 704 |
+
self.model_name = model_name
|
| 705 |
+
self._prepare_model_specific_config()
|
| 706 |
+
|
| 707 |
+
if self.config.output_type is not None:
|
| 708 |
+
if self.config.output_type not in ALL_OUTPUT_TYPES:
|
| 709 |
+
raise ValueError(f"Got invalid output_type '{self.config.output_type}', must be in '{','.join(ALL_OUTPUT_TYPES)}'")
|
| 710 |
+
self.OUTPUT_TYPE = self.config.output_type
|
| 711 |
+
|
| 712 |
+
if self.config.dataset_path is not None:
|
| 713 |
+
self.DATASET_PATH = self.config.dataset_path
|
| 714 |
+
|
| 715 |
+
if self.config.dataset_name is not None:
|
| 716 |
+
self.DATASET_NAME = self.config.dataset_name
|
| 717 |
+
|
| 718 |
+
self._prepare_metric_and_aggregation()
|
| 719 |
+
|
| 720 |
+
self.download(self.config.dataset_kwargs)
|
| 721 |
+
self._training_docs = None
|
| 722 |
+
self._fewshot_docs = None
|
| 723 |
+
|
| 724 |
+
if self.config.filter_list is not None:
|
| 725 |
+
self._filters = []
|
| 726 |
+
for filter_config in self.config.filter_list:
|
| 727 |
+
for filter_pipeline in filter_config:
|
| 728 |
+
filter_name = filter_config["name"]
|
| 729 |
+
filter_functions = filter_config["filter"]
|
| 730 |
+
components = []
|
| 731 |
+
for function in filter_functions:
|
| 732 |
+
kwargs = {key: function[key] for key in function if key != "function"}
|
| 733 |
+
components.append([function["function"], kwargs])
|
| 734 |
+
filter_pipeline = build_filter_ensemble(filter_name, components)
|
| 735 |
+
self._filters.append(filter_pipeline)
|
| 736 |
+
else:
|
| 737 |
+
self._filters = [build_filter_ensemble("none", [["take_first", None]])]
|
| 738 |
+
if self.config.fewshot_config is not None:
|
| 739 |
+
self.sampler = samplers.get_sampler(self.config.fewshot_config.get("sampler", "default") if self.config.fewshot_config else "default")(list(self.fewshot_docs()), self, rnd=random.Random(1234))
|
| 740 |
+
|
| 741 |
+
if self.has_test_docs():
|
| 742 |
+
self.task_docs = self.test_docs()
|
| 743 |
+
elif self.has_validation_docs():
|
| 744 |
+
self.task_docs = self.validation_docs()
|
| 745 |
+
else:
|
| 746 |
+
assert False, f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!"
|
| 747 |
+
|
| 748 |
+
# Test One Doc
|
| 749 |
+
self.features = list(self.task_docs.features.keys())
|
| 750 |
+
self.multiple_input = 0
|
| 751 |
+
self.multiple_target = 0
|
| 752 |
+
test_doc = self.task_docs[0]
|
| 753 |
+
test_text = self.doc_to_text(test_doc)
|
| 754 |
+
test_target = self.doc_to_target(test_doc)
|
| 755 |
+
|
| 756 |
+
if self.config.doc_to_choice is not None:
|
| 757 |
+
test_choice = self.doc_to_choice(test_doc)
|
| 758 |
+
if type(test_choice) is not list:
|
| 759 |
+
eval_logger.error("doc_to_choice must return list")
|
| 760 |
+
else:
|
| 761 |
+
num_choice = len(test_choice)
|
| 762 |
+
|
| 763 |
+
if type(test_text) is int:
|
| 764 |
+
self.multiple_input = num_choice
|
| 765 |
+
else:
|
| 766 |
+
test_choice = None
|
| 767 |
+
|
| 768 |
+
if type(test_target) is list:
|
| 769 |
+
self.multiple_target = len(test_target)
|
| 770 |
+
else:
|
| 771 |
+
if (type(test_target) is int) and (test_choice is not None):
|
| 772 |
+
test_target = test_choice[test_target]
|
| 773 |
+
else:
|
| 774 |
+
test_target = str(test_target)
|
| 775 |
+
|
| 776 |
+
if test_choice is not None:
|
| 777 |
+
check_choices = test_choice
|
| 778 |
+
else:
|
| 779 |
+
check_choices = [test_target]
|
| 780 |
+
if self.config.doc_to_choice is not None:
|
| 781 |
+
for choice in check_choices:
|
| 782 |
+
choice_has_whitespace = True if choice[0].isspace() else False
|
| 783 |
+
delimiter_has_whitespace = True if self.config.target_delimiter.rstrip() != self.config.target_delimiter else False
|
| 784 |
+
|
| 785 |
+
if delimiter_has_whitespace and choice_has_whitespace:
|
| 786 |
+
eval_logger.warning(f'Both target_delimiter and target choice: "{choice}" have whitespace')
|
| 787 |
+
elif (not delimiter_has_whitespace) and (not choice_has_whitespace):
|
| 788 |
+
eval_logger.warning(f'Both target_delimiter "{self.config.target_delimiter}" and target choice: "{choice}" do not have whitespace, ignore if the language you are evaluating on does not require/use whitespace')
|
| 789 |
+
|
| 790 |
+
def _prepare_model_specific_config(self):
|
| 791 |
+
self.lmms_eval_specific_kwargs = self.config.lmms_eval_specific_kwargs
|
| 792 |
+
if self.lmms_eval_specific_kwargs is not None:
|
| 793 |
+
if self.model_name in self.lmms_eval_specific_kwargs:
|
| 794 |
+
self.lmms_eval_specific_kwargs = self.lmms_eval_specific_kwargs[self.model_name]
|
| 795 |
+
elif "default" in self.lmms_eval_specific_kwargs:
|
| 796 |
+
self.lmms_eval_specific_kwargs.update(self.lmms_eval_specific_kwargs.get("default", {}))
|
| 797 |
+
elif "dataset" in self.lmms_eval_specific_kwargs:
|
| 798 |
+
self.lmms_eval_specific_kwargs.update(self.lmms_eval_specific_kwargs.get("dataset", {}))
|
| 799 |
+
|
| 800 |
+
self.model_specific_target_kwargs = self.config.model_specific_target_kwargs
|
| 801 |
+
if self.model_specific_target_kwargs is not None:
|
| 802 |
+
if self.model_name in self.model_specific_target_kwargs:
|
| 803 |
+
self.model_specific_target_kwargs = self.model_specific_target_kwargs[self.model_name]
|
| 804 |
+
else:
|
| 805 |
+
self.model_specific_target_kwargs = self.model_specific_target_kwargs.get("default", None)
|
| 806 |
+
self.model_specific_generation_kwargs = self.config.model_specific_generation_kwargs
|
| 807 |
+
if self.model_specific_generation_kwargs is not None:
|
| 808 |
+
if self.model_name in self.model_specific_generation_kwargs:
|
| 809 |
+
self.model_specific_generation_kwargs = self.model_specific_generation_kwargs[self.model_name]
|
| 810 |
+
else:
|
| 811 |
+
self.model_specific_generation_kwargs = self.model_specific_generation_kwargs.get("default", {})
|
| 812 |
+
|
| 813 |
+
self.config.generation_kwargs.update(self.model_specific_generation_kwargs)
|
| 814 |
+
|
| 815 |
+
def _prepare_metric_and_aggregation(self):
|
| 816 |
+
self._metric_fn_list = {}
|
| 817 |
+
self._metric_fn_kwargs = {}
|
| 818 |
+
self._aggregation_list = {}
|
| 819 |
+
self._higher_is_better = {}
|
| 820 |
+
|
| 821 |
+
if self.config.metric_list is None:
|
| 822 |
+
# TODO: handle this in TaskConfig.__post_init__ ?
|
| 823 |
+
_metric_list = DEFAULT_METRIC_REGISTRY[self.config.output_type]
|
| 824 |
+
|
| 825 |
+
for metric_name in _metric_list:
|
| 826 |
+
self._metric_fn_list[metric_name] = METRIC_REGISTRY[metric_name]
|
| 827 |
+
self._metric_fn_kwargs[metric_name] = {}
|
| 828 |
+
self._aggregation_list[metric_name] = get_metric_aggregation(metric_name)
|
| 829 |
+
self._higher_is_better[metric_name] = is_higher_better(metric_name)
|
| 830 |
+
else:
|
| 831 |
+
for metric_config in self.config.metric_list:
|
| 832 |
+
assert "metric" in metric_config
|
| 833 |
+
metric_name = metric_config["metric"]
|
| 834 |
+
kwargs = {key: metric_config[key] for key in metric_config if key not in ["metric", "aggregation", "higher_is_better"]}
|
| 835 |
+
|
| 836 |
+
if self.config.process_results is not None:
|
| 837 |
+
self._metric_fn_list[metric_name] = None
|
| 838 |
+
self._metric_fn_kwargs[metric_name] = {}
|
| 839 |
+
elif callable(metric_name):
|
| 840 |
+
metric_fn = metric_name.__call__
|
| 841 |
+
metric_name = metric_name.__name__
|
| 842 |
+
self._metric_fn_list[metric_name] = metric_fn
|
| 843 |
+
self._metric_fn_kwargs[metric_name] = kwargs
|
| 844 |
+
else:
|
| 845 |
+
self._metric_fn_list[metric_name] = METRIC_REGISTRY[metric_name]
|
| 846 |
+
self._metric_fn_kwargs[metric_name] = kwargs
|
| 847 |
+
|
| 848 |
+
if "aggregation" in metric_config:
|
| 849 |
+
agg_name = metric_config["aggregation"]
|
| 850 |
+
if type(agg_name) == str:
|
| 851 |
+
self._aggregation_list[metric_name] = get_aggregation(agg_name)
|
| 852 |
+
elif callable(agg_name):
|
| 853 |
+
self._aggregation_list[metric_name] = metric_config["aggregation"]
|
| 854 |
+
else:
|
| 855 |
+
INV_AGG_REGISTRY = {v: k for k, v in AGGREGATION_REGISTRY.items()}
|
| 856 |
+
metric_agg = get_metric_aggregation(metric_name)
|
| 857 |
+
eval_logger.warning(f"[Task: {self._config.task}] metric {metric_name} is defined, but aggregation is not. " f"using default " f"aggregation={INV_AGG_REGISTRY[metric_agg]}")
|
| 858 |
+
self._aggregation_list[metric_name] = metric_agg
|
| 859 |
+
|
| 860 |
+
if "higher_is_better" in metric_config:
|
| 861 |
+
self._higher_is_better[metric_name] = metric_config["higher_is_better"]
|
| 862 |
+
else:
|
| 863 |
+
eval_logger.warning(f"[Task: {self._config.task}] metric {metric_name} is defined, but higher_is_better is not. " f"using default " f"higher_is_better={is_higher_better(metric_name)}")
|
| 864 |
+
self._higher_is_better[metric_name] = is_higher_better(metric_name)
|
| 865 |
+
|
| 866 |
+
@retry(stop=(stop_after_attempt(5) | stop_after_delay(60)), wait=wait_fixed(2))
|
| 867 |
+
def download(self, dataset_kwargs=None) -> None:
|
| 868 |
+
# If the dataset is a video dataset,
|
| 869 |
+
# Recursively search whether their is a zip and unzip it to the huggingface home
|
| 870 |
+
download_config = DownloadConfig()
|
| 871 |
+
download_config.max_retries = dataset_kwargs.get("max_retries", 10) if dataset_kwargs is not None else 10
|
| 872 |
+
download_config.num_proc = dataset_kwargs.get("num_proc", 8) if dataset_kwargs is not None else 8
|
| 873 |
+
download_config.local_files_only = dataset_kwargs.get("local_files_only", False) if dataset_kwargs is not None else False
|
| 874 |
+
if dataset_kwargs is not None:
|
| 875 |
+
if "From_YouTube" in dataset_kwargs:
|
| 876 |
+
|
| 877 |
+
def _download_from_youtube(path):
|
| 878 |
+
try:
|
| 879 |
+
for video in tqdm(self.all_dataset[split]):
|
| 880 |
+
video_id = video["videoID"]
|
| 881 |
+
target_path = os.path.join(path, f"{video_id}.mp4")
|
| 882 |
+
assert shutil.which("yt-dlp") is not None, "yt-dlp must be installed and available in the system's PATH"
|
| 883 |
+
command = f"yt-dlp -o {target_path} -f mp4 https://www.youtube.com/watch?v={video_id}"
|
| 884 |
+
subprocess.run(command, shell=True)
|
| 885 |
+
with open(os.path.join(cache_path, f"{task}_download_status.json"), "w") as f:
|
| 886 |
+
f.write(json.dumps({task: "downloaded"}))
|
| 887 |
+
except Exception as e:
|
| 888 |
+
eval_logger.error(f"Error while downloading {task} data: {e}")
|
| 889 |
+
with open(os.path.join(cache_path, f"{task}_download_status.json"), "w") as f:
|
| 890 |
+
f.write(json.dumps({task: "not downloaded"}))
|
| 891 |
+
|
| 892 |
+
hf_home = os.getenv("HF_HOME", "~/.cache/huggingface/")
|
| 893 |
+
accelerator = Accelerator()
|
| 894 |
+
if accelerator.is_main_process:
|
| 895 |
+
dataset_kwargs.pop("From_YouTube")
|
| 896 |
+
assert "load_from_disk" not in dataset_kwargs, "load_from_disk must not be True when From_YouTube is True"
|
| 897 |
+
self.all_dataset = datasets.load_dataset(
|
| 898 |
+
path=self.DATASET_PATH,
|
| 899 |
+
name=self.DATASET_NAME,
|
| 900 |
+
download_mode=datasets.DownloadMode.REUSE_DATASET_IF_EXISTS,
|
| 901 |
+
**dataset_kwargs if dataset_kwargs is not None else {},
|
| 902 |
+
)
|
| 903 |
+
dataset_kwargs["From_YouTube"] = True
|
| 904 |
+
cache_path = snapshot_download(repo_id=self.DATASET_PATH, repo_type="dataset") # download_parquet
|
| 905 |
+
split = vars(self.config)["test_split"]
|
| 906 |
+
task = vars(self.config)["task"]
|
| 907 |
+
|
| 908 |
+
video_path = os.path.join(hf_home, task)
|
| 909 |
+
if os.path.exists(os.path.join(cache_path, f"{task}_download_status.json")):
|
| 910 |
+
download_status = json.load(open(os.path.join(cache_path, f"{task}_download_status.json"), "r"))
|
| 911 |
+
if download_status[task] == "downloaded":
|
| 912 |
+
eval_logger.info(f"Data for {task} already download!")
|
| 913 |
+
else:
|
| 914 |
+
eval_logger.info(f"Start downloading YouTube data to {video_path}...")
|
| 915 |
+
_download_from_youtube(video_path)
|
| 916 |
+
else:
|
| 917 |
+
eval_logger.info(f"Start downloading YouTube data to {video_path}...")
|
| 918 |
+
_download_from_youtube(video_path)
|
| 919 |
+
|
| 920 |
+
accelerator.wait_for_everyone()
|
| 921 |
+
if "builder_script" in dataset_kwargs:
|
| 922 |
+
builder_script = dataset_kwargs["builder_script"]
|
| 923 |
+
self.DATASET_PATH = os.path.join(cache_path, builder_script)
|
| 924 |
+
dataset_kwargs.pop("builder_script")
|
| 925 |
+
|
| 926 |
+
downloaded_video_ids = [i.split(".mp4")[0] for i in os.listdir(os.path.expanduser(video_path)) if i.endswith(".mp4")]
|
| 927 |
+
# Filtered the existing dataset with the downloaded video ids
|
| 928 |
+
self.dataset = datasets.DatasetDict({split: self.all_dataset[split].filter(lambda x: x["videoID"] in downloaded_video_ids)})
|
| 929 |
+
|
| 930 |
+
self.dataset_no_image = self.dataset
|
| 931 |
+
dataset_kwargs.pop("From_YouTube")
|
| 932 |
+
return
|
| 933 |
+
|
| 934 |
+
if "video" in dataset_kwargs and dataset_kwargs["video"]:
|
| 935 |
+
hf_home = os.getenv("HF_HOME", "~/.cache/huggingface/")
|
| 936 |
+
hf_home = os.path.expanduser(hf_home)
|
| 937 |
+
cache_dir = dataset_kwargs["cache_dir"]
|
| 938 |
+
cache_dir = os.path.join(hf_home, cache_dir)
|
| 939 |
+
accelerator = Accelerator()
|
| 940 |
+
if accelerator.is_main_process:
|
| 941 |
+
force_download = dataset_kwargs.get("force_download", False)
|
| 942 |
+
force_unzip = dataset_kwargs.get("force_unzip", False)
|
| 943 |
+
revision = dataset_kwargs.get("revision", "main")
|
| 944 |
+
create_link = dataset_kwargs.get("create_link", False)
|
| 945 |
+
cache_path = snapshot_download(repo_id=self.DATASET_PATH, revision=revision, repo_type="dataset", force_download=force_download, etag_timeout=60)
|
| 946 |
+
zip_files = glob(os.path.join(cache_path, "**/*.zip"), recursive=True)
|
| 947 |
+
tar_files = glob(os.path.join(cache_path, "**/*.tar*"), recursive=True)
|
| 948 |
+
|
| 949 |
+
def unzip_video_data(zip_file):
|
| 950 |
+
import os
|
| 951 |
+
import zipfile
|
| 952 |
+
|
| 953 |
+
with zipfile.ZipFile(zip_file, "r") as zip_ref:
|
| 954 |
+
for file_info in zip_ref.infolist():
|
| 955 |
+
target_path = os.path.join(cache_dir, file_info.filename)
|
| 956 |
+
if not os.path.exists(target_path):
|
| 957 |
+
zip_ref.extract(file_info, cache_dir)
|
| 958 |
+
else:
|
| 959 |
+
eval_logger.info(f"Skipping existing file: {target_path}")
|
| 960 |
+
|
| 961 |
+
eval_logger.info(f"Extracted all files from {zip_file} to {cache_dir}")
|
| 962 |
+
|
| 963 |
+
def untar_video_data(tar_file):
|
| 964 |
+
import tarfile
|
| 965 |
+
|
| 966 |
+
with tarfile.open(tar_file, "r") as tar_ref:
|
| 967 |
+
tar_ref.extractall(cache_dir)
|
| 968 |
+
eval_logger.info(f"Extracted all files from {tar_file} to {cache_dir}")
|
| 969 |
+
|
| 970 |
+
def concat_tar_parts(tar_parts, output_tar):
|
| 971 |
+
with open(output_tar, "wb") as out_tar:
|
| 972 |
+
from tqdm import tqdm
|
| 973 |
+
|
| 974 |
+
for part in tqdm(sorted(tar_parts)):
|
| 975 |
+
with open(part, "rb") as part_file:
|
| 976 |
+
out_tar.write(part_file.read())
|
| 977 |
+
eval_logger.info(f"Concatenated parts {tar_parts} into {output_tar}")
|
| 978 |
+
|
| 979 |
+
# Unzip zip files if needed
|
| 980 |
+
if force_unzip or (not os.path.exists(cache_dir) and len(zip_files) > 0):
|
| 981 |
+
for zip_file in zip_files:
|
| 982 |
+
unzip_video_data(zip_file)
|
| 983 |
+
|
| 984 |
+
# Concatenate and extract tar files if needed
|
| 985 |
+
if force_unzip or (not os.path.exists(cache_dir) and len(tar_files) > 0):
|
| 986 |
+
tar_parts_dict = {}
|
| 987 |
+
|
| 988 |
+
# Group tar parts together
|
| 989 |
+
for tar_file in tar_files:
|
| 990 |
+
base_name = tar_file.split(".tar")[0]
|
| 991 |
+
if base_name not in tar_parts_dict:
|
| 992 |
+
tar_parts_dict[base_name] = []
|
| 993 |
+
tar_parts_dict[base_name].append(tar_file)
|
| 994 |
+
|
| 995 |
+
# Concatenate and untar split parts
|
| 996 |
+
for base_name, parts in tar_parts_dict.items():
|
| 997 |
+
eval_logger.info(f"Extracting following tar files: {parts}")
|
| 998 |
+
output_tar = base_name + ".tar"
|
| 999 |
+
if not os.path.exists(output_tar):
|
| 1000 |
+
eval_logger.info(f"Start concatenating tar files")
|
| 1001 |
+
|
| 1002 |
+
concat_tar_parts(parts, output_tar)
|
| 1003 |
+
eval_logger.info(f"Finish concatenating tar files")
|
| 1004 |
+
|
| 1005 |
+
if not os.path.exists(os.path.join(cache_dir, os.path.basename(base_name))):
|
| 1006 |
+
untar_video_data(output_tar)
|
| 1007 |
+
|
| 1008 |
+
# Link cache_path to cache_dir if needed.
|
| 1009 |
+
if create_link:
|
| 1010 |
+
if not os.path.exists(cache_dir) or os.path.islink(cache_dir):
|
| 1011 |
+
if os.path.islink(cache_dir):
|
| 1012 |
+
os.remove(cache_dir)
|
| 1013 |
+
eval_logger.info(f"Removed existing symbolic link: {cache_dir}")
|
| 1014 |
+
# Create a new symbolic link
|
| 1015 |
+
os.symlink(cache_path, cache_dir)
|
| 1016 |
+
eval_logger.info(f"Symbolic link created successfully: {cache_path} -> {cache_dir}")
|
| 1017 |
+
|
| 1018 |
+
accelerator.wait_for_everyone()
|
| 1019 |
+
dataset_kwargs.pop("cache_dir")
|
| 1020 |
+
dataset_kwargs.pop("video")
|
| 1021 |
+
|
| 1022 |
+
if "builder_script" in dataset_kwargs:
|
| 1023 |
+
builder_script = dataset_kwargs["builder_script"]
|
| 1024 |
+
self.DATASET_PATH = os.path.join(cache_path, builder_script)
|
| 1025 |
+
dataset_kwargs.pop("builder_script")
|
| 1026 |
+
|
| 1027 |
+
if "force_download" in dataset_kwargs:
|
| 1028 |
+
dataset_kwargs.pop("force_download")
|
| 1029 |
+
|
| 1030 |
+
if "force_unzip" in dataset_kwargs:
|
| 1031 |
+
dataset_kwargs.pop("force_unzip")
|
| 1032 |
+
|
| 1033 |
+
if "local_files_only" in dataset_kwargs:
|
| 1034 |
+
dataset_kwargs.pop("local_files_only")
|
| 1035 |
+
|
| 1036 |
+
if "create_link" in dataset_kwargs:
|
| 1037 |
+
dataset_kwargs.pop("create_link")
|
| 1038 |
+
|
| 1039 |
+
if dataset_kwargs is not None and "load_from_disk" in dataset_kwargs and dataset_kwargs["load_from_disk"]:
|
| 1040 |
+
dataset_kwargs.pop("load_from_disk")
|
| 1041 |
+
# using local task in offline environment, need to process the online dataset into local format via
|
| 1042 |
+
# `ds = load_datasets("lmms-lab/MMMU")`
|
| 1043 |
+
self.dataset = datasets.load_from_disk(path=self.DATASET_PATH, name=self.DATASET_NAME)
|
| 1044 |
+
else:
|
| 1045 |
+
self.dataset = datasets.load_dataset(
|
| 1046 |
+
path=self.DATASET_PATH,
|
| 1047 |
+
name=self.DATASET_NAME,
|
| 1048 |
+
download_mode=datasets.DownloadMode.REUSE_DATASET_IF_EXISTS,
|
| 1049 |
+
download_config=download_config,
|
| 1050 |
+
**dataset_kwargs if dataset_kwargs is not None else {},
|
| 1051 |
+
)
|
| 1052 |
+
|
| 1053 |
+
if self.config.process_docs is not None:
|
| 1054 |
+
for split in self.dataset:
|
| 1055 |
+
if split in [self.config.training_split, self.config.validation_split, self.config.test_split, self.config.fewshot_split]:
|
| 1056 |
+
self.dataset[split] = self.config.process_docs(self.dataset[split])
|
| 1057 |
+
|
| 1058 |
+
# copy dataset, remove image features
|
| 1059 |
+
self.dataset_no_image = self.dataset.copy()
|
| 1060 |
+
for doc_name in self.dataset_no_image:
|
| 1061 |
+
remove_cols = []
|
| 1062 |
+
features = self.dataset_no_image[doc_name].features
|
| 1063 |
+
# If it is an Image instance or a Sequence of Image instance. Remove it
|
| 1064 |
+
for feature in features:
|
| 1065 |
+
if isinstance(features[feature], Image):
|
| 1066 |
+
remove_cols.append(feature)
|
| 1067 |
+
elif isinstance(features[feature], Sequence) and isinstance(features[feature].feature, Image):
|
| 1068 |
+
remove_cols.append(feature)
|
| 1069 |
+
elif isinstance(features[feature], Audio):
|
| 1070 |
+
remove_cols.append(feature)
|
| 1071 |
+
for remove_col in remove_cols:
|
| 1072 |
+
self.dataset_no_image[doc_name] = self.dataset_no_image[doc_name].remove_columns(remove_col)
|
| 1073 |
+
|
| 1074 |
+
def has_training_docs(self) -> bool:
|
| 1075 |
+
if self.config.training_split is not None:
|
| 1076 |
+
return True
|
| 1077 |
+
else:
|
| 1078 |
+
return False
|
| 1079 |
+
|
| 1080 |
+
def has_validation_docs(self) -> bool:
|
| 1081 |
+
if self.config.validation_split is not None:
|
| 1082 |
+
return True
|
| 1083 |
+
else:
|
| 1084 |
+
return False
|
| 1085 |
+
|
| 1086 |
+
def has_test_docs(self) -> bool:
|
| 1087 |
+
if self.config.test_split is not None:
|
| 1088 |
+
return True
|
| 1089 |
+
else:
|
| 1090 |
+
return False
|
| 1091 |
+
|
| 1092 |
+
def training_docs(self) -> datasets.Dataset:
|
| 1093 |
+
if self.has_training_docs():
|
| 1094 |
+
return self.dataset[self.config.training_split]
|
| 1095 |
+
|
| 1096 |
+
def validation_docs(self) -> datasets.Dataset:
|
| 1097 |
+
if self.has_validation_docs():
|
| 1098 |
+
return self.dataset[self.config.validation_split]
|
| 1099 |
+
|
| 1100 |
+
def validation_docs_no_media(self) -> datasets.Dataset:
|
| 1101 |
+
if self.has_validation_docs():
|
| 1102 |
+
return self.dataset_no_image[self.config.validation_split]
|
| 1103 |
+
|
| 1104 |
+
def test_docs(self) -> datasets.Dataset:
|
| 1105 |
+
if self.has_test_docs():
|
| 1106 |
+
return self.dataset[self.config.test_split]
|
| 1107 |
+
|
| 1108 |
+
def test_docs_no_media(self) -> datasets.Dataset:
|
| 1109 |
+
if self.has_test_docs():
|
| 1110 |
+
return self.dataset_no_image[self.config.test_split]
|
| 1111 |
+
|
| 1112 |
+
@property
|
| 1113 |
+
def eval_docs_no_media(self) -> Union[datasets.Dataset, List[dict]]:
|
| 1114 |
+
if self.has_test_docs():
|
| 1115 |
+
return self.test_docs_no_media()
|
| 1116 |
+
elif self.has_validation_docs():
|
| 1117 |
+
return self.validation_docs_no_media()
|
| 1118 |
+
else:
|
| 1119 |
+
raise ValueError(f"Task dataset (path={self.DATASET_PATH}, name={self.DATASET_NAME}) must have valid or test docs!")
|
| 1120 |
+
|
| 1121 |
+
def fewshot_docs(self):
|
| 1122 |
+
if self.config.fewshot_split is not None:
|
| 1123 |
+
return self.dataset[self.config.fewshot_split]
|
| 1124 |
+
else:
|
| 1125 |
+
if (self.config.num_fewshot is not None) and (self.config.num_fewshot > 0):
|
| 1126 |
+
eval_logger.warning(f"Task '{self.config.task}': " "num_fewshot > 0 but fewshot_split is None. " "using preconfigured rule.")
|
| 1127 |
+
return super().fewshot_docs()
|
| 1128 |
+
|
| 1129 |
+
@utils.positional_deprecated
|
| 1130 |
+
def fewshot_context(
|
| 1131 |
+
self,
|
| 1132 |
+
doc: str,
|
| 1133 |
+
num_fewshot: int,
|
| 1134 |
+
system_instruction: Optional[str] = None,
|
| 1135 |
+
apply_chat_template: bool = False,
|
| 1136 |
+
fewshot_as_multiturn: bool = False,
|
| 1137 |
+
chat_template: Optional[Callable] = None,
|
| 1138 |
+
is_multimodal: bool = False,
|
| 1139 |
+
) -> str:
|
| 1140 |
+
"""Returns a fewshot context string that is made up of a prepended description
|
| 1141 |
+
(if provided), the `num_fewshot` number of examples, and an appended prompt example.
|
| 1142 |
+
|
| 1143 |
+
:param doc: str
|
| 1144 |
+
The document as returned from training_docs, validation_docs, or test_docs.
|
| 1145 |
+
:param num_fewshot: int
|
| 1146 |
+
The number of fewshot examples to provide in the returned context string.
|
| 1147 |
+
:param system_instruction: str
|
| 1148 |
+
System instruction to be applied to the prompt.
|
| 1149 |
+
:param apply_chat_template: bool
|
| 1150 |
+
Whether to apply the chat template to the fewshot context.
|
| 1151 |
+
:param fewshot_as_multiturn: bool
|
| 1152 |
+
Whether to provide the fewshot examples as a multiturn conversation or a single user turn.
|
| 1153 |
+
:param chat_template:
|
| 1154 |
+
callable (from lm.apply_chat_template) that takes in a list[Dict] chat transcript and renders it into a string.
|
| 1155 |
+
:returns: str
|
| 1156 |
+
The fewshot context.
|
| 1157 |
+
"""
|
| 1158 |
+
|
| 1159 |
+
if apply_chat_template:
|
| 1160 |
+
labeled_examples = []
|
| 1161 |
+
else:
|
| 1162 |
+
labeled_examples = ""
|
| 1163 |
+
|
| 1164 |
+
# get task description
|
| 1165 |
+
if description := self.config.description:
|
| 1166 |
+
description = utils.apply_template(self.config.description, doc)
|
| 1167 |
+
|
| 1168 |
+
# create system prompt based on the provided system instruction and description
|
| 1169 |
+
if system_instruction is not None and description:
|
| 1170 |
+
system_prompt = f"{system_instruction}{self.sampler.fewshot_delimiter}{description}"
|
| 1171 |
+
elif system_instruction is not None:
|
| 1172 |
+
system_prompt = system_instruction
|
| 1173 |
+
elif description:
|
| 1174 |
+
system_prompt = description
|
| 1175 |
+
else:
|
| 1176 |
+
system_prompt = ""
|
| 1177 |
+
|
| 1178 |
+
# add system prompt if specified
|
| 1179 |
+
if system_prompt:
|
| 1180 |
+
if apply_chat_template:
|
| 1181 |
+
labeled_examples.append({"role": "system", "content": system_prompt})
|
| 1182 |
+
else:
|
| 1183 |
+
labeled_examples = system_prompt
|
| 1184 |
+
|
| 1185 |
+
# if few-shot - append examples after the system prompt
|
| 1186 |
+
if num_fewshot > 0:
|
| 1187 |
+
if is_multimodal is False:
|
| 1188 |
+
if apply_chat_template:
|
| 1189 |
+
labeled_examples.extend(self.sampler.get_chat_context(doc, num_fewshot, fewshot_as_multiturn))
|
| 1190 |
+
else:
|
| 1191 |
+
labeled_examples += self.sampler.get_context(doc, num_fewshot)
|
| 1192 |
+
else:
|
| 1193 |
+
if apply_chat_template:
|
| 1194 |
+
labeled_examples_text, labeled_examples_multimodal = self.sampler.get_multimodal_chat_context(doc, num_fewshot, fewshot_as_multiturn)
|
| 1195 |
+
labeled_examples.extend(labeled_examples_text)
|
| 1196 |
+
else:
|
| 1197 |
+
labeled_examples_text, labeled_examples_multimodal = self.sampler.get_multimodal_context(doc, num_fewshot)
|
| 1198 |
+
labeled_examples += labeled_examples_text
|
| 1199 |
+
|
| 1200 |
+
example = self.doc_to_text(doc)
|
| 1201 |
+
if is_multimodal is False:
|
| 1202 |
+
if apply_chat_template:
|
| 1203 |
+
if self.multiple_input:
|
| 1204 |
+
return chat_template(labeled_examples)
|
| 1205 |
+
if isinstance(example, str):
|
| 1206 |
+
self.append_target_question(labeled_examples, example, fewshot_as_multiturn)
|
| 1207 |
+
# for loglikelihood create a list of questions with appended choices
|
| 1208 |
+
elif isinstance(example, list):
|
| 1209 |
+
labeled_examples_list = []
|
| 1210 |
+
# copy chat history for each example and append the answer
|
| 1211 |
+
for ex in example:
|
| 1212 |
+
chat = copy.deepcopy(labeled_examples)
|
| 1213 |
+
self.append_target_question(chat, ex, fewshot_as_multiturn)
|
| 1214 |
+
labeled_examples_list.append(chat_template(chat))
|
| 1215 |
+
return labeled_examples_list
|
| 1216 |
+
# if example is an integer, append the choice or convert to string
|
| 1217 |
+
elif isinstance(example, int):
|
| 1218 |
+
if self.config.doc_to_choice is not None:
|
| 1219 |
+
choices = self.doc_to_choice(doc)
|
| 1220 |
+
self.append_target_question(labeled_examples, choices[example], fewshot_as_multiturn)
|
| 1221 |
+
else:
|
| 1222 |
+
self.append_target_question(labeled_examples, str(example), fewshot_as_multiturn)
|
| 1223 |
+
# return lm.apply_chat_template(labeled_examples)
|
| 1224 |
+
return chat_template(labeled_examples)
|
| 1225 |
+
else:
|
| 1226 |
+
if self.multiple_input:
|
| 1227 |
+
return labeled_examples
|
| 1228 |
+
if isinstance(example, str):
|
| 1229 |
+
return labeled_examples + example
|
| 1230 |
+
elif isinstance(example, list):
|
| 1231 |
+
return [labeled_examples + ex for ex in example]
|
| 1232 |
+
elif isinstance(example, int):
|
| 1233 |
+
if self.config.doc_to_choice is not None:
|
| 1234 |
+
choices = self.doc_to_choice(doc)
|
| 1235 |
+
return labeled_examples + choices[example]
|
| 1236 |
+
else:
|
| 1237 |
+
return labeled_examples + str(example)
|
| 1238 |
+
else:
|
| 1239 |
+
if apply_chat_template:
|
| 1240 |
+
raise NotImplementedError("Multimodal chat template not implemented yet")
|
| 1241 |
+
else:
|
| 1242 |
+
if self.multiple_input:
|
| 1243 |
+
return labeled_examples + "<image> " + example, labeled_examples_multimodal
|
| 1244 |
+
if isinstance(example, str):
|
| 1245 |
+
return labeled_examples + "<image> " + example, labeled_examples_multimodal
|
| 1246 |
+
else:
|
| 1247 |
+
raise NotImplementedError("Multimodal not implemented yet")
|
| 1248 |
+
# elif isinstance(example, list):
|
| 1249 |
+
# return [labeled_examples + ex for ex in example]
|
| 1250 |
+
# elif isinstance(example, int):
|
| 1251 |
+
# if self.config.doc_to_choice is not None:
|
| 1252 |
+
# choices = self.doc_to_choice(doc)
|
| 1253 |
+
# return labeled_examples + choices[example], labeled_examples_multimodal
|
| 1254 |
+
# else:
|
| 1255 |
+
# return labeled_examples + str(example), labeled_examples_multimodal
|
| 1256 |
+
|
| 1257 |
+
def apply_filters(self) -> Optional[List[Instance]]:
|
| 1258 |
+
"""Iterates over FilterEnsembles and applies them to instances"""
|
| 1259 |
+
if hasattr(self, "_filters"):
|
| 1260 |
+
for f in self._filters:
|
| 1261 |
+
f.apply(self._instances, self.task_docs)
|
| 1262 |
+
else:
|
| 1263 |
+
eval_logger.warning("No filter defined, passing through instances")
|
| 1264 |
+
return self._instances
|
| 1265 |
+
|
| 1266 |
+
def should_decontaminate(self):
|
| 1267 |
+
return self.config.should_decontaminate
|
| 1268 |
+
|
| 1269 |
+
def doc_to_decontamination_query(self, doc):
|
| 1270 |
+
if self.config.should_decontaminate:
|
| 1271 |
+
if self.config.doc_to_decontamination_query is None:
|
| 1272 |
+
return self.doc_to_text(doc)
|
| 1273 |
+
else:
|
| 1274 |
+
doc_to_decontamination_query = self.config.doc_to_decontamination_query
|
| 1275 |
+
if doc_to_decontamination_query in self.features:
|
| 1276 |
+
return doc[doc_to_decontamination_query]
|
| 1277 |
+
elif callable(doc_to_decontamination_query):
|
| 1278 |
+
return doc_to_decontamination_query(doc)
|
| 1279 |
+
else:
|
| 1280 |
+
return ast.literal_eval(utils.apply_template(self.config.doc_to_decontamination_query, doc))
|
| 1281 |
+
|
| 1282 |
+
def _process_doc(self, doc):
|
| 1283 |
+
"""
|
| 1284 |
+
Override this to process (detokenize, strip, replace, etc.) individual
|
| 1285 |
+
documents. This can be used in a map over documents of a data split.
|
| 1286 |
+
E.g. `map(self._process_doc, self.dataset["validation"])`
|
| 1287 |
+
|
| 1288 |
+
:return: dict
|
| 1289 |
+
The processed version of the specified `doc`.
|
| 1290 |
+
"""
|
| 1291 |
+
return doc
|
| 1292 |
+
|
| 1293 |
+
def doc_to_text(self, doc):
|
| 1294 |
+
doc_to_text = self.config.doc_to_text
|
| 1295 |
+
|
| 1296 |
+
if type(doc_to_text) == int:
|
| 1297 |
+
return doc_to_text
|
| 1298 |
+
elif type(doc_to_text) == str:
|
| 1299 |
+
if doc_to_text in self.features:
|
| 1300 |
+
# if self.config.doc_to_choice is not None:
|
| 1301 |
+
# return self.doc_to_choice(doc)[doc[doc_to_text]]
|
| 1302 |
+
# else:
|
| 1303 |
+
return doc[doc_to_text]
|
| 1304 |
+
else:
|
| 1305 |
+
text_string = utils.apply_template(doc_to_text, doc)
|
| 1306 |
+
if text_string.isdigit() and self._config.doc_to_choice is not None:
|
| 1307 |
+
return ast.literal_eval(text_string)
|
| 1308 |
+
else:
|
| 1309 |
+
return text_string
|
| 1310 |
+
elif callable(doc_to_text):
|
| 1311 |
+
return (
|
| 1312 |
+
doc_to_text(doc, self.lmms_eval_specific_kwargs)
|
| 1313 |
+
if self.lmms_eval_specific_kwargs is not None
|
| 1314 |
+
else doc_to_text(
|
| 1315 |
+
doc,
|
| 1316 |
+
)
|
| 1317 |
+
)
|
| 1318 |
+
# Used when applying a Promptsource template
|
| 1319 |
+
elif hasattr(doc_to_text, "apply"):
|
| 1320 |
+
applied_prompt = doc_to_text.apply(doc)
|
| 1321 |
+
if len(applied_prompt) == 2:
|
| 1322 |
+
return applied_prompt[0]
|
| 1323 |
+
else:
|
| 1324 |
+
eval_logger.warning("Applied prompt returns empty string")
|
| 1325 |
+
return self.config.fewshot_delimiter
|
| 1326 |
+
else:
|
| 1327 |
+
print(type(doc_to_text))
|
| 1328 |
+
raise TypeError
|
| 1329 |
+
|
| 1330 |
+
def doc_to_target(self, doc: dict) -> Union[int, str, list]:
|
| 1331 |
+
doc_to_target = self.config.doc_to_target
|
| 1332 |
+
|
| 1333 |
+
if type(doc_to_target) == int:
|
| 1334 |
+
return doc_to_target
|
| 1335 |
+
elif type(doc_to_target) == str:
|
| 1336 |
+
if doc_to_target in self.features:
|
| 1337 |
+
# if self.config.doc_to_choice is not None:
|
| 1338 |
+
# return self.doc_to_choice(doc)[doc[doc_to_target]]
|
| 1339 |
+
# else:
|
| 1340 |
+
return doc[doc_to_target]
|
| 1341 |
+
else:
|
| 1342 |
+
target_string = utils.apply_template(doc_to_target, doc)
|
| 1343 |
+
if target_string.isdigit() and self._config.doc_to_choice is not None:
|
| 1344 |
+
return ast.literal_eval(target_string)
|
| 1345 |
+
elif len(target_string) >= 2 and (target_string[0] == "[") and (target_string[-1] == "]"):
|
| 1346 |
+
try:
|
| 1347 |
+
return ast.literal_eval(target_string)
|
| 1348 |
+
except (SyntaxError, ValueError):
|
| 1349 |
+
return target_string
|
| 1350 |
+
else:
|
| 1351 |
+
return target_string
|
| 1352 |
+
elif type(doc_to_target) == list:
|
| 1353 |
+
return doc_to_target
|
| 1354 |
+
elif callable(doc_to_target):
|
| 1355 |
+
return doc_to_target(doc, self.model_specific_target_kwargs) if self.model_specific_target_kwargs is not None else doc_to_target(doc)
|
| 1356 |
+
# Used when applying a Promptsource template
|
| 1357 |
+
elif hasattr(doc_to_target, "apply"):
|
| 1358 |
+
applied_prompt = doc_to_target.apply(doc)
|
| 1359 |
+
if len(applied_prompt) == 2:
|
| 1360 |
+
return applied_prompt[1]
|
| 1361 |
+
else:
|
| 1362 |
+
eval_logger.warning("Applied prompt returns empty string")
|
| 1363 |
+
return self.config.fewshot_delimiter
|
| 1364 |
+
else:
|
| 1365 |
+
raise TypeError
|
| 1366 |
+
|
| 1367 |
+
def doc_to_visual(self, doc: dict) -> Union[int, str, list]:
|
| 1368 |
+
self.config.doc_to_visual
|
| 1369 |
+
if type(self.config.doc_to_visual) == str:
|
| 1370 |
+
assert self.config.doc_to_visual in self.features
|
| 1371 |
+
# Single image. Still return a list for consistency.
|
| 1372 |
+
return [doc[self.config.doc_to_visual]]
|
| 1373 |
+
elif callable(self.config.doc_to_visual):
|
| 1374 |
+
return (
|
| 1375 |
+
self.config.doc_to_visual(doc, self.lmms_eval_specific_kwargs)
|
| 1376 |
+
if self.lmms_eval_specific_kwargs is not None and len(inspect.signature(self.config.doc_to_visual).parameters) == 2
|
| 1377 |
+
else self.config.doc_to_visual(
|
| 1378 |
+
doc,
|
| 1379 |
+
)
|
| 1380 |
+
)
|
| 1381 |
+
else:
|
| 1382 |
+
# eval_logger.warning("Note that doc_to_visual was called but not set in config. Please check if this is a text-only task.")
|
| 1383 |
+
return self.config.doc_to_visual
|
| 1384 |
+
|
| 1385 |
+
def doc_to_choice(self, doc: Any) -> List[str]:
|
| 1386 |
+
if self.config.doc_to_choice is None:
|
| 1387 |
+
eval_logger.error("Note that doc_to_choice was called but not set in config.")
|
| 1388 |
+
else:
|
| 1389 |
+
doc_to_choice = self.config.doc_to_choice
|
| 1390 |
+
|
| 1391 |
+
if type(doc_to_choice) == str:
|
| 1392 |
+
if doc_to_choice in self.features:
|
| 1393 |
+
return doc[doc_to_choice]
|
| 1394 |
+
else:
|
| 1395 |
+
return ast.literal_eval(utils.apply_template(doc_to_choice, doc))
|
| 1396 |
+
elif type(doc_to_choice) == list:
|
| 1397 |
+
return doc_to_choice
|
| 1398 |
+
elif type(doc_to_choice) == dict:
|
| 1399 |
+
return list(doc_to_choice.values())
|
| 1400 |
+
elif callable(doc_to_choice):
|
| 1401 |
+
return doc_to_choice(doc)
|
| 1402 |
+
elif hasattr(doc_to_choice, "get_answer_choices_list"):
|
| 1403 |
+
return doc_to_choice.get_answer_choices_list(doc)
|
| 1404 |
+
else:
|
| 1405 |
+
raise TypeError
|
| 1406 |
+
|
| 1407 |
+
def construct_requests(self, doc_id: int, ctx: str, **kwargs) -> Union[List[Instance], Instance]:
|
| 1408 |
+
split = kwargs.get("metadata").get("split")
|
| 1409 |
+
# kwargs.pop("split")
|
| 1410 |
+
if self.OUTPUT_TYPE == "loglikelihood":
|
| 1411 |
+
arguments = (ctx, self.doc_to_target, self.doc_to_visual, doc_id, self.config.task, split)
|
| 1412 |
+
elif self.OUTPUT_TYPE == "multiple_choice":
|
| 1413 |
+
doc = self.dataset[split][doc_id]
|
| 1414 |
+
choices = self.doc_to_choice(doc)
|
| 1415 |
+
target_delimiter = self.config.target_delimiter
|
| 1416 |
+
if self.multiple_input:
|
| 1417 |
+
# If there are multiple inputs, choices are placed in the ctx
|
| 1418 |
+
cont = self.doc_to_target(doc)
|
| 1419 |
+
arguments = [(ctx, f"{target_delimiter}{cont}", self.doc_to_visual, doc_id, self.config.task, split) for ctx in choices]
|
| 1420 |
+
else:
|
| 1421 |
+
# Otherwise they are placed in the continuation
|
| 1422 |
+
arguments = [(ctx, f"{target_delimiter}{cont}", self.doc_to_visual, doc_id, self.config.task, split) for cont in choices]
|
| 1423 |
+
request_list = [
|
| 1424 |
+
Instance(
|
| 1425 |
+
request_type="loglikelihood",
|
| 1426 |
+
# doc=doc,
|
| 1427 |
+
arguments=arg,
|
| 1428 |
+
idx=i,
|
| 1429 |
+
**kwargs,
|
| 1430 |
+
)
|
| 1431 |
+
for i, arg in enumerate(arguments)
|
| 1432 |
+
]
|
| 1433 |
+
# TODO: we should raise a warning telling users this will at most ~2x runtime.
|
| 1434 |
+
if "acc_mutual_info" in self._metric_fn_list.keys():
|
| 1435 |
+
# if we are calculating multiple choice accuracy
|
| 1436 |
+
# using mutual information instead of raw loglikelihood as metric, need unconditional lls.
|
| 1437 |
+
|
| 1438 |
+
# here mutual info refers to calculating
|
| 1439 |
+
# log(P(choice|ctx) / P(choice)) = log(P(choice|ctx)) - log(P(choice))
|
| 1440 |
+
# in other words normalizing by subtracting the unconditional logprob of each choice.
|
| 1441 |
+
request_list.extend(
|
| 1442 |
+
[
|
| 1443 |
+
Instance(
|
| 1444 |
+
request_type="loglikelihood",
|
| 1445 |
+
# doc=doc,
|
| 1446 |
+
arguments=("", "{}".format(choice)),
|
| 1447 |
+
idx=i,
|
| 1448 |
+
**kwargs,
|
| 1449 |
+
)
|
| 1450 |
+
for i, choice in enumerate(choices)
|
| 1451 |
+
]
|
| 1452 |
+
)
|
| 1453 |
+
return request_list
|
| 1454 |
+
|
| 1455 |
+
elif self.OUTPUT_TYPE == "generate_until":
|
| 1456 |
+
arguments = (ctx, copy.deepcopy(self.config.generation_kwargs), self.doc_to_visual, doc_id, self.config.task, split)
|
| 1457 |
+
elif self.OUTPUT_TYPE == "generate_until_multi_round":
|
| 1458 |
+
arguments = (ctx, copy.deepcopy(self.config.generation_kwargs), self.doc_to_visual, partial(self.config.doc_to_text, lmms_eval_specific_kwargs=self.lmms_eval_specific_kwargs), doc_id, self.config.task, split)
|
| 1459 |
+
return Instance(request_type=self.OUTPUT_TYPE, arguments=arguments, idx=0, **kwargs)
|
| 1460 |
+
|
| 1461 |
+
# TODO: we add a full_docs interface here for some evaluations that needs to access the full datasets during process_results function. we may have better ways to handle this.
|
| 1462 |
+
@retry(stop=(stop_after_attempt(5) | stop_after_delay(1200)), wait=wait_fixed(2))
|
| 1463 |
+
def process_results(self, doc, results, full_docs=None):
|
| 1464 |
+
if self.OUTPUT_TYPE == "generate_until":
|
| 1465 |
+
if isinstance(results, list) and isinstance(results[0], list):
|
| 1466 |
+
results = [res.strip() for res in results[0]]
|
| 1467 |
+
else:
|
| 1468 |
+
results = [res.strip() for res in results]
|
| 1469 |
+
|
| 1470 |
+
kwargs = {}
|
| 1471 |
+
if full_docs is not None:
|
| 1472 |
+
kwargs["full_docs"] = full_docs
|
| 1473 |
+
if callable(self.config.process_results):
|
| 1474 |
+
return self.config.process_results(doc, results, **kwargs)
|
| 1475 |
+
|
| 1476 |
+
result_dict = {}
|
| 1477 |
+
use_metric = list(self._metric_fn_list.keys())
|
| 1478 |
+
if self.OUTPUT_TYPE == "loglikelihood":
|
| 1479 |
+
ll, is_greedy = results
|
| 1480 |
+
return {
|
| 1481 |
+
**({"perplexity": ll} if "perplexity" in use_metric else {}),
|
| 1482 |
+
**({"acc": int(is_greedy)} if "acc" in use_metric else {}),
|
| 1483 |
+
}
|
| 1484 |
+
elif self.OUTPUT_TYPE == "multiple_choice":
|
| 1485 |
+
lls, is_greedy = zip(*results)
|
| 1486 |
+
|
| 1487 |
+
# retrieve choices in List[str] form, to compute choice lengths, etc.
|
| 1488 |
+
choices = self.doc_to_choice(doc)
|
| 1489 |
+
completion_len = np.array([float(len(i)) for i in choices])
|
| 1490 |
+
|
| 1491 |
+
if 2 * len(choices) == len(lls) and "acc_mutual_info" in self._metric_fn_list.keys():
|
| 1492 |
+
# then we are doing mutual info.
|
| 1493 |
+
# this stores the "dryrun" / unconditional answer loglikelihoods
|
| 1494 |
+
lls_unconditional = lls[1::2]
|
| 1495 |
+
assert len(lls_unconditional) == len(choices)
|
| 1496 |
+
# and this stores our "regular" conditional loglikelihoods
|
| 1497 |
+
lls = lls[::2]
|
| 1498 |
+
|
| 1499 |
+
# Warning :
|
| 1500 |
+
# Here may be different from original lm-eval
|
| 1501 |
+
# since we return the actual loss in many model loglikelihood
|
| 1502 |
+
# we just use the argmin here
|
| 1503 |
+
pred = np.argmin(lls)
|
| 1504 |
+
pred_norm = np.argmin(lls / completion_len)
|
| 1505 |
+
|
| 1506 |
+
if self.multiple_input:
|
| 1507 |
+
gold = self.doc_to_text(doc)
|
| 1508 |
+
else:
|
| 1509 |
+
gold = self.doc_to_target(doc)
|
| 1510 |
+
|
| 1511 |
+
gold_index_error = False
|
| 1512 |
+
if type(gold) is list:
|
| 1513 |
+
gold = [i if i < len(choices) else -100 for i in gold]
|
| 1514 |
+
if -100 in gold:
|
| 1515 |
+
gold_index_error = True
|
| 1516 |
+
else:
|
| 1517 |
+
if type(gold) is int:
|
| 1518 |
+
gold = gold if gold < len(choices) else -100
|
| 1519 |
+
elif type(gold) is str:
|
| 1520 |
+
gold = choices.index(gold) if gold in choices else -100
|
| 1521 |
+
|
| 1522 |
+
if gold == -100:
|
| 1523 |
+
gold_index_error = True
|
| 1524 |
+
|
| 1525 |
+
if gold_index_error:
|
| 1526 |
+
eval_logger.warning(f"Label index was not in within range of available choices," f"Sample:\n\n{doc}\n\n")
|
| 1527 |
+
|
| 1528 |
+
if self.multiple_target:
|
| 1529 |
+
acc = 1.0 if pred in gold else 0.0
|
| 1530 |
+
acc_norm = 1.0 if pred_norm in gold else 0.0
|
| 1531 |
+
exact_match = int(any([is_greedy[i] if i != -100 else 0 for i in gold]))
|
| 1532 |
+
else:
|
| 1533 |
+
acc = 1.0 if pred == gold else 0.0
|
| 1534 |
+
acc_norm = 1.0 if pred_norm == gold else 0.0
|
| 1535 |
+
# TODO: this gets score of 0 on arc_challenge for pythia-70m. need to test that this works properly
|
| 1536 |
+
exact_match = int(is_greedy[gold]) if gold != -100 else 0
|
| 1537 |
+
|
| 1538 |
+
result_dict = {
|
| 1539 |
+
**({"acc": acc} if "acc" in use_metric else {}),
|
| 1540 |
+
**({"f1": (gold, pred)} if "f1" in use_metric else {}),
|
| 1541 |
+
**({"mcc": (gold, pred)} if "mcc" in use_metric else {}),
|
| 1542 |
+
**({"acc_norm": acc_norm} if "acc_norm" in use_metric else {}),
|
| 1543 |
+
**({"exact_match": exact_match} if "exact_match" in use_metric else {}),
|
| 1544 |
+
}
|
| 1545 |
+
|
| 1546 |
+
if "acc_mutual_info" in use_metric:
|
| 1547 |
+
lls_mutual_info = [ll_c - ll_u for ll_c, ll_u in zip(lls, lls_unconditional)]
|
| 1548 |
+
acc_mutual_info = 1.0 if np.argmax(lls_mutual_info) == gold else 0.0
|
| 1549 |
+
result_dict["acc_mutual_info"] = acc_mutual_info
|
| 1550 |
+
|
| 1551 |
+
elif "generate_until" in self.OUTPUT_TYPE:
|
| 1552 |
+
gold = self.doc_to_target(doc)
|
| 1553 |
+
result = [res.strip() for res in results]
|
| 1554 |
+
if self.config.doc_to_choice is not None:
|
| 1555 |
+
# If you set doc_to_choice,
|
| 1556 |
+
# it assumes that doc_to_target returns a number.
|
| 1557 |
+
choices = self.doc_to_choice(doc)
|
| 1558 |
+
gold = choices[gold]
|
| 1559 |
+
# we expect multiple_targets to be a list.
|
| 1560 |
+
elif self.multiple_target:
|
| 1561 |
+
gold = list(gold)
|
| 1562 |
+
# elif type(gold) != type(result):
|
| 1563 |
+
# # cast gold to the same type as result
|
| 1564 |
+
# gold = type(result)(gold)
|
| 1565 |
+
|
| 1566 |
+
for metric in self._metric_fn_list.keys():
|
| 1567 |
+
if self.multiple_target and metric != "anls":
|
| 1568 |
+
# in the case where we have multiple targets,
|
| 1569 |
+
# return true if any are true
|
| 1570 |
+
# TODO: this may break for multipLe_target, non zero-or-1 metrics
|
| 1571 |
+
scores = []
|
| 1572 |
+
if not isinstance(gold, list):
|
| 1573 |
+
# sometimes, a multiple_target dataset has exceptions where one doc has only one string answer
|
| 1574 |
+
# print(gold)
|
| 1575 |
+
gold = [gold]
|
| 1576 |
+
for gold_option in gold:
|
| 1577 |
+
try:
|
| 1578 |
+
result_score = self._metric_fn_list[metric](
|
| 1579 |
+
references=[gold_option],
|
| 1580 |
+
predictions=result,
|
| 1581 |
+
**self._metric_fn_kwargs[metric],
|
| 1582 |
+
)
|
| 1583 |
+
except TypeError: # TODO: this is hacky and I don't want to do it
|
| 1584 |
+
result_score = self._metric_fn_list[metric]([gold_option, result])
|
| 1585 |
+
if isinstance(result_score, dict):
|
| 1586 |
+
# TODO: this handles the case where HF evaluate returns a dict.
|
| 1587 |
+
result_score = result_score[metric]
|
| 1588 |
+
scores.append(result_score)
|
| 1589 |
+
if any(scores):
|
| 1590 |
+
result_score = 1.0
|
| 1591 |
+
else:
|
| 1592 |
+
result_score = 0.0
|
| 1593 |
+
else:
|
| 1594 |
+
if not isinstance(gold, list):
|
| 1595 |
+
gold = [gold]
|
| 1596 |
+
try:
|
| 1597 |
+
result_score = self._metric_fn_list[metric](
|
| 1598 |
+
references=gold,
|
| 1599 |
+
predictions=result,
|
| 1600 |
+
**self._metric_fn_kwargs[metric],
|
| 1601 |
+
)
|
| 1602 |
+
except TypeError: # needed for now in order to use a different interface between our own metrics and HF Evaluate metrics
|
| 1603 |
+
result_score = self._metric_fn_list[metric]([gold, result])
|
| 1604 |
+
if isinstance(result_score, dict):
|
| 1605 |
+
# TODO: this handles the case where HF evaluate returns a dict.
|
| 1606 |
+
result_score = result_score[metric]
|
| 1607 |
+
result_dict[metric] = result_score
|
| 1608 |
+
else:
|
| 1609 |
+
raise ValueError(
|
| 1610 |
+
f"Passed invalid output_type '{self.OUTPUT_TYPE}' ! Please use one of ",
|
| 1611 |
+
"'loglikelihood','generate_until', 'generate_until_multi_round', or 'multiple_choice'",
|
| 1612 |
+
)
|
| 1613 |
+
|
| 1614 |
+
return result_dict
|
| 1615 |
+
|
| 1616 |
+
def aggregation(self):
|
| 1617 |
+
return self._aggregation_list
|
| 1618 |
+
|
| 1619 |
+
def higher_is_better(self):
|
| 1620 |
+
return self._higher_is_better
|
| 1621 |
+
|
| 1622 |
+
def get_config(self, key: str) -> Any:
|
| 1623 |
+
return getattr(self._config, key, None)
|
| 1624 |
+
|
| 1625 |
+
@property
|
| 1626 |
+
def task_name(self) -> Any:
|
| 1627 |
+
return getattr(self.config, "task", None)
|
| 1628 |
+
|
| 1629 |
+
def __repr__(self):
|
| 1630 |
+
return f"ConfigurableTask(task_name={getattr(self.config, 'task', None)}," f"output_type={self.OUTPUT_TYPE}," f"num_fewshot={getattr(self.config, 'num_fewshot', None)}," f"num_samples={len(self.eval_docs)})"
|
LaViDa/eval/lmms_eval/caching/__init__.py
ADDED
|
File without changes
|
LaViDa/eval/lmms_eval/caching/cache.py
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import hashlib
|
| 2 |
+
import os
|
| 3 |
+
|
| 4 |
+
import dill
|
| 5 |
+
|
| 6 |
+
from lmms_eval.loggers.utils import _handle_non_serializable
|
| 7 |
+
from lmms_eval.utils import eval_logger
|
| 8 |
+
|
| 9 |
+
MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
|
| 10 |
+
|
| 11 |
+
OVERRIDE_PATH = os.getenv("LM_HARNESS_CACHE_PATH")
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
PATH = OVERRIDE_PATH if OVERRIDE_PATH else f"{MODULE_DIR}/.cache"
|
| 15 |
+
|
| 16 |
+
# This should be sufficient for uniqueness
|
| 17 |
+
HASH_INPUT = "EleutherAI-lm-evaluation-harness"
|
| 18 |
+
|
| 19 |
+
HASH_PREFIX = hashlib.sha256(HASH_INPUT.encode("utf-8")).hexdigest()
|
| 20 |
+
|
| 21 |
+
FILE_SUFFIX = f".{HASH_PREFIX}.pickle"
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def load_from_cache(file_name):
|
| 25 |
+
try:
|
| 26 |
+
path = f"{PATH}/{file_name}{FILE_SUFFIX}"
|
| 27 |
+
|
| 28 |
+
with open(path, "rb") as file:
|
| 29 |
+
cached_task_dict = dill.loads(file.read())
|
| 30 |
+
return cached_task_dict
|
| 31 |
+
|
| 32 |
+
except Exception:
|
| 33 |
+
eval_logger.debug(f"{file_name} is not cached, generating...")
|
| 34 |
+
pass
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def save_to_cache(file_name, obj):
|
| 38 |
+
if not os.path.exists(PATH):
|
| 39 |
+
os.mkdir(PATH)
|
| 40 |
+
|
| 41 |
+
file_path = f"{PATH}/{file_name}{FILE_SUFFIX}"
|
| 42 |
+
|
| 43 |
+
serializable_obj = []
|
| 44 |
+
|
| 45 |
+
for item in obj:
|
| 46 |
+
sub_serializable_obj = []
|
| 47 |
+
for subitem in item:
|
| 48 |
+
if hasattr(subitem, "arguments"): # we need to handle the arguments specially since doc_to_visual is callable method and not serializable
|
| 49 |
+
serializable_arguments = tuple(arg if not callable(arg) else None for arg in subitem.arguments)
|
| 50 |
+
subitem.arguments = serializable_arguments
|
| 51 |
+
sub_serializable_obj.append(_handle_non_serializable(subitem))
|
| 52 |
+
serializable_obj.append(sub_serializable_obj)
|
| 53 |
+
|
| 54 |
+
eval_logger.debug(f"Saving {file_path} to cache...")
|
| 55 |
+
with open(file_path, "wb") as file:
|
| 56 |
+
file.write(dill.dumps(serializable_obj))
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
# NOTE the "key" param is to allow for flexibility
|
| 60 |
+
def delete_cache(key: str = ""):
|
| 61 |
+
files = os.listdir(PATH)
|
| 62 |
+
|
| 63 |
+
for file in files:
|
| 64 |
+
if file.startswith(key) and file.endswith(FILE_SUFFIX):
|
| 65 |
+
file_path = f"{PATH}/{file}"
|
| 66 |
+
os.unlink(file_path)
|
LaViDa/eval/lmms_eval/eval_server.py
ADDED
|
File without changes
|
LaViDa/eval/lmms_eval/evaluator.py
ADDED
|
@@ -0,0 +1,654 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import collections
|
| 2 |
+
import inspect
|
| 3 |
+
import itertools
|
| 4 |
+
import json
|
| 5 |
+
import os
|
| 6 |
+
import random
|
| 7 |
+
import sys
|
| 8 |
+
import time
|
| 9 |
+
from collections import defaultdict
|
| 10 |
+
from dataclasses import dataclass
|
| 11 |
+
from typing import List, Optional, Union
|
| 12 |
+
|
| 13 |
+
import numpy as np
|
| 14 |
+
import torch
|
| 15 |
+
import torch.distributed as dist
|
| 16 |
+
from datasets import Image, Sequence
|
| 17 |
+
from loguru import logger as eval_logger
|
| 18 |
+
from tqdm import tqdm
|
| 19 |
+
|
| 20 |
+
import lmms_eval.api
|
| 21 |
+
import lmms_eval.api.metrics
|
| 22 |
+
import lmms_eval.api.registry
|
| 23 |
+
from lmms_eval.evaluator_utils import (
|
| 24 |
+
consolidate_group_results,
|
| 25 |
+
consolidate_results,
|
| 26 |
+
get_sample_size,
|
| 27 |
+
get_subtask_list,
|
| 28 |
+
get_task_list,
|
| 29 |
+
prepare_print_tasks,
|
| 30 |
+
print_writeout,
|
| 31 |
+
run_task_tests,
|
| 32 |
+
)
|
| 33 |
+
from lmms_eval.loggers.evaluation_tracker import EvaluationTracker
|
| 34 |
+
from lmms_eval.models import get_model
|
| 35 |
+
from lmms_eval.tasks import TaskManager, get_task_dict
|
| 36 |
+
from lmms_eval.utils import (
|
| 37 |
+
create_iterator,
|
| 38 |
+
get_datetime_str,
|
| 39 |
+
get_git_commit_hash,
|
| 40 |
+
handle_non_serializable,
|
| 41 |
+
hash_string,
|
| 42 |
+
make_table,
|
| 43 |
+
positional_deprecated,
|
| 44 |
+
run_task_tests,
|
| 45 |
+
simple_parse_args_string,
|
| 46 |
+
)
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
@positional_deprecated
|
| 50 |
+
def simple_evaluate(
|
| 51 |
+
model,
|
| 52 |
+
model_args: Optional[Union[str, dict]] = None,
|
| 53 |
+
tasks: Optional[List[Union[str, dict, object]]] = None,
|
| 54 |
+
num_fewshot: Optional[int] = None,
|
| 55 |
+
batch_size: Optional[Union[int, str]] = None,
|
| 56 |
+
max_batch_size: Optional[int] = None,
|
| 57 |
+
device: Optional[str] = None,
|
| 58 |
+
use_cache: Optional[str] = None,
|
| 59 |
+
cache_requests: bool = False,
|
| 60 |
+
rewrite_requests_cache: bool = False,
|
| 61 |
+
delete_requests_cache: bool = False,
|
| 62 |
+
limit: Optional[Union[int, float]] = None,
|
| 63 |
+
bootstrap_iters: int = 100000,
|
| 64 |
+
check_integrity: bool = False,
|
| 65 |
+
write_out: bool = False,
|
| 66 |
+
log_samples: bool = True,
|
| 67 |
+
evaluation_tracker: Optional[EvaluationTracker] = None,
|
| 68 |
+
system_instruction: Optional[str] = None,
|
| 69 |
+
apply_chat_template: bool = False,
|
| 70 |
+
fewshot_as_multiturn: bool = False,
|
| 71 |
+
gen_kwargs: Optional[str] = None,
|
| 72 |
+
task_manager: Optional[TaskManager] = None,
|
| 73 |
+
verbosity: str = "INFO",
|
| 74 |
+
predict_only: bool = False,
|
| 75 |
+
random_seed: int = 0,
|
| 76 |
+
numpy_random_seed: int = 1234,
|
| 77 |
+
torch_random_seed: int = 1234,
|
| 78 |
+
fewshot_random_seed: int = 1234,
|
| 79 |
+
datetime_str: str = get_datetime_str(),
|
| 80 |
+
cli_args=None,
|
| 81 |
+
):
|
| 82 |
+
"""Instantiate and evaluate a model on a list of tasks.
|
| 83 |
+
|
| 84 |
+
:param model: Union[str, LM]
|
| 85 |
+
Name of model or LM object, see lm_eval.models.get_model
|
| 86 |
+
:param model_args: Optional[str, dict]
|
| 87 |
+
String or dict arguments for each model class, see LM.create_from_arg_string and LM.create_from_arg_object.
|
| 88 |
+
Ignored if `model` argument is a LM object.
|
| 89 |
+
:param tasks: list[Union[str, dict, Task]]
|
| 90 |
+
List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.
|
| 91 |
+
:param num_fewshot: int
|
| 92 |
+
Number of examples in few-shot context
|
| 93 |
+
:param batch_size: int or str, optional
|
| 94 |
+
Batch size for model
|
| 95 |
+
:param max_batch_size: int, optional
|
| 96 |
+
Maximal batch size to try with automatic batch size detection
|
| 97 |
+
:param device: str, optional
|
| 98 |
+
PyTorch device (e.g. "cpu" or "cuda:0") for running models
|
| 99 |
+
:param use_cache: str, optional
|
| 100 |
+
A path to a sqlite db file for caching model responses. `None` if not caching.
|
| 101 |
+
:param cache_requests: bool, optional
|
| 102 |
+
Speed up evaluation by caching the building of dataset requests. `None` if not caching.
|
| 103 |
+
:param rewrite_requests_cache: bool, optional
|
| 104 |
+
Rewrites all of the request cache if set to `True`. `None` if not desired.
|
| 105 |
+
:param delete_requests_cache: bool, optional
|
| 106 |
+
Deletes all of the request cache if set to `True`. `None` if not desired.
|
| 107 |
+
:param limit: int or float, optional
|
| 108 |
+
Limit the number of examples per task (only use this for testing), If <1, limit is a percentage of the total number of examples.
|
| 109 |
+
:param bootstrap_iters:
|
| 110 |
+
Number of iterations for bootstrap statistics, used when calculating stderrs. set to 0 for no stderr calculations to be performed.
|
| 111 |
+
:param check_integrity: bool
|
| 112 |
+
Whether to run the relevant part of the test suite for the tasks
|
| 113 |
+
:param write_out: bool
|
| 114 |
+
If True, write out an example document and model input for checking task integrity
|
| 115 |
+
:param log_samples: bool
|
| 116 |
+
If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis
|
| 117 |
+
:param system_instruction: str
|
| 118 |
+
System instruction to be applied to the prompt
|
| 119 |
+
:param apply_chat_template: bool
|
| 120 |
+
If True, apply chat template to the prompt
|
| 121 |
+
:param fewshot_as_multiturn: bool
|
| 122 |
+
Whether to provide the fewshot examples as a multiturn conversation or a single user turn.
|
| 123 |
+
:param gen_kwargs: str
|
| 124 |
+
String arguments for model generation
|
| 125 |
+
Ignored for all tasks with loglikelihood output_type
|
| 126 |
+
:param predict_only: bool
|
| 127 |
+
If true only model outputs will be generated and returned. Metrics will not be evaluated
|
| 128 |
+
:param random_seed: int
|
| 129 |
+
Random seed for python's random module. If set to None, the seed will not be set.
|
| 130 |
+
:param numpy_random_seed: int
|
| 131 |
+
Random seed for numpy. If set to None, the seed will not be set.
|
| 132 |
+
:param torch_random_seed: int
|
| 133 |
+
Random seed for torch. If set to None, the seed will not be set.
|
| 134 |
+
:param fewshot_random_seed: int
|
| 135 |
+
Random seed for fewshot sampler random generator. If set to None, the seed of generator will be set to None.
|
| 136 |
+
|
| 137 |
+
:return
|
| 138 |
+
Dictionary of results
|
| 139 |
+
"""
|
| 140 |
+
seed_message = []
|
| 141 |
+
if random_seed is not None:
|
| 142 |
+
# See https://github.com/EleutherAI/lm-evaluation-harness/pull/1412
|
| 143 |
+
seed_message.append(f"Setting random seed to {random_seed}")
|
| 144 |
+
random.seed(random_seed)
|
| 145 |
+
|
| 146 |
+
if numpy_random_seed is not None:
|
| 147 |
+
seed_message.append(f"Setting numpy seed to {numpy_random_seed}")
|
| 148 |
+
np.random.seed(numpy_random_seed)
|
| 149 |
+
|
| 150 |
+
if torch_random_seed is not None:
|
| 151 |
+
seed_message.append(f"Setting torch manual seed to {torch_random_seed}")
|
| 152 |
+
torch.manual_seed(torch_random_seed)
|
| 153 |
+
|
| 154 |
+
if seed_message:
|
| 155 |
+
eval_logger.info(" | ".join(seed_message))
|
| 156 |
+
|
| 157 |
+
assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names."
|
| 158 |
+
|
| 159 |
+
if gen_kwargs:
|
| 160 |
+
gen_kwargs = simple_parse_args_string(gen_kwargs)
|
| 161 |
+
eval_logger.warning(f"generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.")
|
| 162 |
+
if gen_kwargs == "":
|
| 163 |
+
gen_kwargs = None
|
| 164 |
+
|
| 165 |
+
if model_args is None:
|
| 166 |
+
model_args = ""
|
| 167 |
+
|
| 168 |
+
if task_manager is None:
|
| 169 |
+
task_manager = TaskManager(verbosity, model_name=model)
|
| 170 |
+
|
| 171 |
+
task_dict = get_task_dict(tasks, task_manager)
|
| 172 |
+
|
| 173 |
+
if isinstance(model, str):
|
| 174 |
+
if model_args is None:
|
| 175 |
+
model_args = ""
|
| 176 |
+
lm = lmms_eval.models.get_model(model).create_from_arg_string(
|
| 177 |
+
model_args,
|
| 178 |
+
{
|
| 179 |
+
"batch_size": batch_size,
|
| 180 |
+
"max_batch_size": max_batch_size,
|
| 181 |
+
"device": device,
|
| 182 |
+
},
|
| 183 |
+
)
|
| 184 |
+
elif isinstance(model, lmms_eval.api.model.lmms):
|
| 185 |
+
lm = model
|
| 186 |
+
|
| 187 |
+
# helper function to recursively apply config overrides to leaf subtasks, skipping their constituent groups.
|
| 188 |
+
# (setting of num_fewshot ; bypassing metric calculation ; setting fewshot seed)
|
| 189 |
+
def _adjust_config(task_dict):
|
| 190 |
+
adjusted_task_dict = {}
|
| 191 |
+
for task_name, task_obj in task_dict.items():
|
| 192 |
+
if isinstance(task_obj, dict):
|
| 193 |
+
adjusted_task_dict = {
|
| 194 |
+
**adjusted_task_dict,
|
| 195 |
+
**{task_name: _adjust_config(task_obj)},
|
| 196 |
+
}
|
| 197 |
+
|
| 198 |
+
else:
|
| 199 |
+
task_obj = task_dict[task_name]
|
| 200 |
+
if type(task_obj) == tuple:
|
| 201 |
+
group, task_obj = task_obj
|
| 202 |
+
if task_obj is None:
|
| 203 |
+
continue
|
| 204 |
+
lm.task_dict[task_name] = task_obj.dataset
|
| 205 |
+
if "generate_until" in task_obj.get_config("output_type"):
|
| 206 |
+
if gen_kwargs is not None:
|
| 207 |
+
task_obj.set_config(key="generation_kwargs", value=gen_kwargs, update=True)
|
| 208 |
+
|
| 209 |
+
if predict_only:
|
| 210 |
+
eval_logger.info(f"Processing {task_name} in output-only mode. Metrics will not be calculated!")
|
| 211 |
+
# we have to change the class properties post-hoc. This is pretty hacky.
|
| 212 |
+
task_obj.override_metric(metric_name="bypass")
|
| 213 |
+
|
| 214 |
+
# override tasks' fewshot values to the provided num_fewshot arg value
|
| 215 |
+
# except if tasks have it set to 0 manually in their configs--then we should never overwrite that
|
| 216 |
+
if num_fewshot is not None:
|
| 217 |
+
if (default_num_fewshot := task_obj.get_config("num_fewshot")) == 0:
|
| 218 |
+
eval_logger.info(f"num_fewshot has been set to 0 for {task_name} in its config. Manual configuration will be ignored.")
|
| 219 |
+
else:
|
| 220 |
+
eval_logger.warning(f"Overwriting default num_fewshot of {task_name} from {default_num_fewshot} to {num_fewshot}")
|
| 221 |
+
task_obj.set_config(key="num_fewshot", value=num_fewshot)
|
| 222 |
+
else:
|
| 223 |
+
# if num_fewshot not provided, and the task does not define a default one, default to 0
|
| 224 |
+
if (default_num_fewshot := task_obj.get_config("num_fewshot")) is None:
|
| 225 |
+
task_obj.set_config(key="num_fewshot", value=0)
|
| 226 |
+
# fewshot_random_seed set for tasks, even with a default num_fewshot (e.g. in the YAML file)
|
| 227 |
+
task_obj.set_fewshot_seed(seed=fewshot_random_seed)
|
| 228 |
+
# eval_logger.info(f"Setting fewshot random generator seed to {fewshot_random_seed}")
|
| 229 |
+
|
| 230 |
+
adjusted_task_dict[task_name] = task_obj
|
| 231 |
+
|
| 232 |
+
return adjusted_task_dict
|
| 233 |
+
|
| 234 |
+
task_dict = _adjust_config(task_dict)
|
| 235 |
+
|
| 236 |
+
if check_integrity:
|
| 237 |
+
run_task_tests(task_list=tasks)
|
| 238 |
+
|
| 239 |
+
if evaluation_tracker is not None:
|
| 240 |
+
evaluation_tracker.general_config_tracker.log_experiment_args(
|
| 241 |
+
model_source=model,
|
| 242 |
+
model_args=model_args,
|
| 243 |
+
system_instruction=system_instruction,
|
| 244 |
+
chat_template=lm.chat_template if apply_chat_template else None,
|
| 245 |
+
fewshot_as_multiturn=fewshot_as_multiturn,
|
| 246 |
+
)
|
| 247 |
+
|
| 248 |
+
results = evaluate(
|
| 249 |
+
lm=lm,
|
| 250 |
+
task_dict=task_dict,
|
| 251 |
+
limit=limit,
|
| 252 |
+
cache_requests=cache_requests,
|
| 253 |
+
rewrite_requests_cache=rewrite_requests_cache,
|
| 254 |
+
bootstrap_iters=bootstrap_iters,
|
| 255 |
+
write_out=write_out,
|
| 256 |
+
log_samples=True if predict_only else log_samples,
|
| 257 |
+
system_instruction=system_instruction,
|
| 258 |
+
apply_chat_template=apply_chat_template,
|
| 259 |
+
fewshot_as_multiturn=fewshot_as_multiturn,
|
| 260 |
+
verbosity=verbosity,
|
| 261 |
+
cli_args=cli_args,
|
| 262 |
+
)
|
| 263 |
+
|
| 264 |
+
if lm.rank == 0:
|
| 265 |
+
if isinstance(model, str):
|
| 266 |
+
model_name = model
|
| 267 |
+
elif hasattr(model, "config") and hasattr(model.config, "_name_or_path"):
|
| 268 |
+
model_name = model.config._name_or_path
|
| 269 |
+
else:
|
| 270 |
+
model_name = type(model).__name__
|
| 271 |
+
|
| 272 |
+
# add info about the model and few shot config
|
| 273 |
+
results["config"] = {
|
| 274 |
+
"model": model_name,
|
| 275 |
+
"model_args": model_args,
|
| 276 |
+
}
|
| 277 |
+
# add more detailed model info if available TODO: add model info
|
| 278 |
+
# if isinstance(lm, lm_eval.models.huggingface.HFLM):
|
| 279 |
+
# results["config"].update(lm.get_model_info())
|
| 280 |
+
# add info about execution
|
| 281 |
+
results["config"].update(
|
| 282 |
+
{
|
| 283 |
+
"batch_size": batch_size,
|
| 284 |
+
"batch_sizes": (list(lm.batch_sizes.values()) if hasattr(lm, "batch_sizes") else []),
|
| 285 |
+
"device": device,
|
| 286 |
+
"use_cache": use_cache,
|
| 287 |
+
"limit": limit,
|
| 288 |
+
"bootstrap_iters": bootstrap_iters,
|
| 289 |
+
"gen_kwargs": gen_kwargs,
|
| 290 |
+
"random_seed": random_seed,
|
| 291 |
+
"numpy_seed": numpy_random_seed,
|
| 292 |
+
"torch_seed": torch_random_seed,
|
| 293 |
+
"fewshot_seed": fewshot_random_seed,
|
| 294 |
+
}
|
| 295 |
+
)
|
| 296 |
+
results["git_hash"] = get_git_commit_hash()
|
| 297 |
+
results["date"] = datetime_str
|
| 298 |
+
# add_env_info(results) # additional environment info to results
|
| 299 |
+
# add_tokenizer_info(results, lm) # additional info about tokenizer
|
| 300 |
+
return results
|
| 301 |
+
else:
|
| 302 |
+
return None
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
decontaminate_suffix = "_decontaminate"
|
| 306 |
+
|
| 307 |
+
|
| 308 |
+
@positional_deprecated
|
| 309 |
+
def evaluate(
|
| 310 |
+
lm: "LM",
|
| 311 |
+
task_dict,
|
| 312 |
+
limit: Optional[int] = None,
|
| 313 |
+
cache_requests: bool = False,
|
| 314 |
+
rewrite_requests_cache: bool = False,
|
| 315 |
+
bootstrap_iters: Optional[int] = 100000,
|
| 316 |
+
write_out: bool = False,
|
| 317 |
+
log_samples: bool = True,
|
| 318 |
+
system_instruction: Optional[str] = None,
|
| 319 |
+
apply_chat_template: bool = False,
|
| 320 |
+
fewshot_as_multiturn: bool = False,
|
| 321 |
+
verbosity: str = "INFO",
|
| 322 |
+
cli_args=None,
|
| 323 |
+
):
|
| 324 |
+
"""Instantiate and evaluate a model on a list of tasks.
|
| 325 |
+
|
| 326 |
+
:param lm: obj
|
| 327 |
+
Language Model
|
| 328 |
+
:param task_dict: dict[str, Task]
|
| 329 |
+
Dictionary of tasks. Tasks will be taken to have name type(task).config.task .
|
| 330 |
+
:param limit: int, optional
|
| 331 |
+
Limit the number of examples per task (only use this for testing)
|
| 332 |
+
:param bootstrap_iters:
|
| 333 |
+
Number of iterations for bootstrap statistics, used when calculating stderr. Set to 0 for skipping all stderr calculations.
|
| 334 |
+
:param write_out: bool
|
| 335 |
+
If True, write out an example document and model input for checking task integrity
|
| 336 |
+
:param log_samples: bool
|
| 337 |
+
If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis
|
| 338 |
+
:param system_instruction: str
|
| 339 |
+
System instruction to be applied to the prompt
|
| 340 |
+
:param apply_chat_template: bool
|
| 341 |
+
If True, apply chat template to the prompt
|
| 342 |
+
:param fewshot_as_multiturn: bool
|
| 343 |
+
Whether to provide the fewshot examples as a multiturn conversation or a single user turn.
|
| 344 |
+
:return
|
| 345 |
+
Dictionary of results
|
| 346 |
+
"""
|
| 347 |
+
|
| 348 |
+
# stores the final result for each task, for each metric/filter pair.
|
| 349 |
+
results = collections.defaultdict(dict)
|
| 350 |
+
# Tracks each task's version.
|
| 351 |
+
versions = collections.defaultdict(dict)
|
| 352 |
+
# Tracks the YAML configs of all chosen tasks.
|
| 353 |
+
configs = collections.defaultdict(dict)
|
| 354 |
+
# logs info about each document evaluated.
|
| 355 |
+
samples = collections.defaultdict(list)
|
| 356 |
+
# tracks all Instances/requests a model must generate output on.
|
| 357 |
+
requests = collections.defaultdict(list)
|
| 358 |
+
# Aggregated task scores presented with groups
|
| 359 |
+
results_agg = collections.defaultdict(dict)
|
| 360 |
+
# Aggregated groups scores only
|
| 361 |
+
groups_agg = collections.defaultdict(dict)
|
| 362 |
+
# stores the amount to pad out reqs per req. type so that
|
| 363 |
+
# number of fwd passes per distributed rank is equal
|
| 364 |
+
padding_requests = collections.defaultdict(int)
|
| 365 |
+
# store the hierarchy to do proper ordering
|
| 366 |
+
task_hierarchy = collections.defaultdict(list)
|
| 367 |
+
# store the ordering of tasks and groups
|
| 368 |
+
task_order = collections.defaultdict(int)
|
| 369 |
+
task_group_alias = collections.defaultdict(dict)
|
| 370 |
+
# store num-fewshot value per task
|
| 371 |
+
num_fewshot = collections.defaultdict(int)
|
| 372 |
+
|
| 373 |
+
# get lists of group hierarchy and each type of request
|
| 374 |
+
eval_tasks = get_task_list(task_dict)
|
| 375 |
+
name_to_task = {}
|
| 376 |
+
if not log_samples:
|
| 377 |
+
if not all("bypass" not in getattr(task_output.task, "_metric_fn_list", {}).keys() for task_output in eval_tasks):
|
| 378 |
+
raise ValueError("log_samples must be True for 'bypass' metric-only tasks")
|
| 379 |
+
|
| 380 |
+
for task_output in eval_tasks:
|
| 381 |
+
task: Task = task_output.task
|
| 382 |
+
task_name = task_output.task_name
|
| 383 |
+
task.args = cli_args
|
| 384 |
+
|
| 385 |
+
name_to_task[task_name] = task
|
| 386 |
+
|
| 387 |
+
if type(task) == tuple:
|
| 388 |
+
group_name, task = task
|
| 389 |
+
task_hierarchy[group_name].append(task_name)
|
| 390 |
+
versions[group_name] = "N/A"
|
| 391 |
+
else:
|
| 392 |
+
group_name = None
|
| 393 |
+
task_hierarchy[task_name] = []
|
| 394 |
+
|
| 395 |
+
if task is None:
|
| 396 |
+
continue
|
| 397 |
+
|
| 398 |
+
versions[task_name] = task.VERSION
|
| 399 |
+
configs[task_name] = dict(task.dump_config())
|
| 400 |
+
|
| 401 |
+
if "num_fewshot" in configs[task_name]:
|
| 402 |
+
n_shot = configs[task_name]["num_fewshot"]
|
| 403 |
+
else:
|
| 404 |
+
n_shot = 0
|
| 405 |
+
num_fewshot[task_name] = n_shot
|
| 406 |
+
|
| 407 |
+
if "task_alias" in configs[task_name]:
|
| 408 |
+
task_group_alias[task_name] = configs[task_name]["task_alias"]
|
| 409 |
+
|
| 410 |
+
if ("group_alias" in configs[task_name]) and (group_name not in task_group_alias) and (group_name is not None):
|
| 411 |
+
task_group_alias[group_name] = configs[task_name]["group_alias"]
|
| 412 |
+
|
| 413 |
+
limit = get_sample_size(task, limit)
|
| 414 |
+
task.build_all_requests(
|
| 415 |
+
limit=limit,
|
| 416 |
+
rank=lm.rank,
|
| 417 |
+
world_size=lm.world_size,
|
| 418 |
+
cache_requests=cache_requests, # later we will add them
|
| 419 |
+
rewrite_requests_cache=rewrite_requests_cache,
|
| 420 |
+
system_instruction=system_instruction,
|
| 421 |
+
apply_chat_template=apply_chat_template,
|
| 422 |
+
fewshot_as_multiturn=fewshot_as_multiturn,
|
| 423 |
+
chat_template=getattr(lm, "apply_chat_template") if apply_chat_template else None,
|
| 424 |
+
tokenizer_name=getattr(lm, "tokenizer_name", "") if apply_chat_template else "",
|
| 425 |
+
)
|
| 426 |
+
eval_logger.debug(f"Task: {task_output.task_name}; number of requests on this rank: {len(task._instances)}")
|
| 427 |
+
if write_out:
|
| 428 |
+
print_writeout(task)
|
| 429 |
+
# aggregate Instances by LM method requested to get output.
|
| 430 |
+
for instance in task.instances:
|
| 431 |
+
reqtype = instance.request_type
|
| 432 |
+
requests[reqtype].append(instance)
|
| 433 |
+
|
| 434 |
+
if lm.world_size > 1:
|
| 435 |
+
instances_rnk = torch.tensor(len(task._instances), device=lm.device)
|
| 436 |
+
gathered_item = lm.accelerator.gather(instances_rnk).cpu().detach().numpy().tolist()
|
| 437 |
+
# "multiple_choice" task types dispatch (several) "loglikelihood" request types
|
| 438 |
+
reqtype = "loglikelihood" if task.OUTPUT_TYPE == "multiple_choice" else task.OUTPUT_TYPE
|
| 439 |
+
# compute number of pseudo-batches to pad with (FSDP/DDP require even batches among ranks)
|
| 440 |
+
numpad = max(gathered_item) - gathered_item[lm.rank]
|
| 441 |
+
# todo: may not account for padding in cases like SquadV2 which has multiple req types
|
| 442 |
+
padding_requests[reqtype] += numpad
|
| 443 |
+
|
| 444 |
+
### Run LMM on inputs, get all outputs ###
|
| 445 |
+
# execute each type of request
|
| 446 |
+
for reqtype, reqs in requests.items():
|
| 447 |
+
eval_logger.info("Running {} requests".format(reqtype))
|
| 448 |
+
# create `K` copies of each request `req` based off `K = req.repeats`
|
| 449 |
+
cloned_reqs = []
|
| 450 |
+
for req in reqs:
|
| 451 |
+
cloned_reqs.extend([req] * req.repeats)
|
| 452 |
+
|
| 453 |
+
if (lm.world_size > 1) and (padding_requests[reqtype] > 0):
|
| 454 |
+
for _ in range(padding_requests[reqtype]):
|
| 455 |
+
cloned_reqs.extend([req] * req.repeats)
|
| 456 |
+
|
| 457 |
+
# run requests through model
|
| 458 |
+
resps = getattr(lm, reqtype)(cloned_reqs) # Choiszt run generate until
|
| 459 |
+
|
| 460 |
+
# put responses from model into a list of length K for each request.
|
| 461 |
+
for x, req in zip(resps, cloned_reqs):
|
| 462 |
+
req.resps.append(x)
|
| 463 |
+
|
| 464 |
+
if lm.world_size > 1:
|
| 465 |
+
lm.accelerator.wait_for_everyone()
|
| 466 |
+
|
| 467 |
+
RANK = lm.rank
|
| 468 |
+
WORLD_SIZE = lm.world_size
|
| 469 |
+
### Postprocess outputs ###
|
| 470 |
+
# TODO: del model here, maybe (idea: allow user to specify device of e.g. reward model separately)
|
| 471 |
+
for task_output in eval_tasks:
|
| 472 |
+
task = task_output.task
|
| 473 |
+
task.apply_filters()
|
| 474 |
+
|
| 475 |
+
### Collect values of metrics on all datapoints ###
|
| 476 |
+
# # unpack results and sort back in order and return control to Task
|
| 477 |
+
# TODO: make it possible to use a different metric per filter
|
| 478 |
+
# Pre-process task.instances to group by doc_id
|
| 479 |
+
instances_by_doc_id = collections.defaultdict(list)
|
| 480 |
+
for instance in task.instances:
|
| 481 |
+
instances_by_doc_id[instance.doc_id].append(instance)
|
| 482 |
+
# Sort instances within each group
|
| 483 |
+
for instances in instances_by_doc_id.values():
|
| 484 |
+
instances.sort(key=lambda x: x.idx)
|
| 485 |
+
# iterate over different filters used
|
| 486 |
+
for filter_key in task.instances[0].filtered_resps.keys():
|
| 487 |
+
if not cli_args.process_with_media:
|
| 488 |
+
doc_iterator = create_iterator(enumerate(task.eval_docs_no_media), rank=RANK, limit=int(limit) if limit else None, world_size=WORLD_SIZE)
|
| 489 |
+
else:
|
| 490 |
+
doc_iterator = task.doc_iterator(rank=RANK, limit=limit, world_size=WORLD_SIZE)
|
| 491 |
+
doc_iterator_for_counting = itertools.islice(range(len(task.test_docs())), RANK, limit, WORLD_SIZE) if task.has_test_docs() else itertools.islice(range(len(task.validation_docs())), RANK, limit, WORLD_SIZE)
|
| 492 |
+
total_docs = sum(1 for _ in doc_iterator_for_counting)
|
| 493 |
+
pbar = tqdm(total=total_docs, desc=f"Postprocessing", disable=(RANK != 0))
|
| 494 |
+
for doc_id, doc in doc_iterator:
|
| 495 |
+
requests = instances_by_doc_id[doc_id]
|
| 496 |
+
metrics = task.process_results(doc, [req.filtered_resps[filter_key] for req in requests])
|
| 497 |
+
if log_samples:
|
| 498 |
+
target = task.doc_to_target(doc)
|
| 499 |
+
saved_doc = {}
|
| 500 |
+
for key, value in doc.items():
|
| 501 |
+
# If image is not in key
|
| 502 |
+
if "image" not in key:
|
| 503 |
+
# If audio is also not the value
|
| 504 |
+
if isinstance(value, dict) and "array" in value:
|
| 505 |
+
continue
|
| 506 |
+
else:
|
| 507 |
+
saved_doc[key] = value
|
| 508 |
+
filtered_arguments = []
|
| 509 |
+
for req in requests:
|
| 510 |
+
# check if req.args is a list of tuples, and each item in the list is a serializable object
|
| 511 |
+
for value in req.args:
|
| 512 |
+
if isinstance(value, (str, int, float, bool, list, dict, type(None))):
|
| 513 |
+
filtered_arguments.append(value)
|
| 514 |
+
# else:
|
| 515 |
+
# filtered_arguments.append(_handle_non_serializable(value))
|
| 516 |
+
|
| 517 |
+
example = {
|
| 518 |
+
"doc_id": doc_id,
|
| 519 |
+
"doc": saved_doc,
|
| 520 |
+
"target": target,
|
| 521 |
+
"arguments": filtered_arguments,
|
| 522 |
+
"resps": [req.resps for req in requests],
|
| 523 |
+
"filtered_resps": [req.filtered_resps[filter_key] for req in requests],
|
| 524 |
+
"doc_hash": hash_string(
|
| 525 |
+
json.dumps(
|
| 526 |
+
requests[0].doc,
|
| 527 |
+
indent=2,
|
| 528 |
+
default=handle_non_serializable,
|
| 529 |
+
ensure_ascii=False,
|
| 530 |
+
)
|
| 531 |
+
),
|
| 532 |
+
"prompt_hash": hash_string(requests[0].arguments[0]),
|
| 533 |
+
"target_hash": hash_string(str(target)),
|
| 534 |
+
}
|
| 535 |
+
example.update(metrics)
|
| 536 |
+
task_output.logged_samples.append(example)
|
| 537 |
+
for metric, value in metrics.items():
|
| 538 |
+
task_output.sample_metrics[(metric, filter_key)].append(value)
|
| 539 |
+
pbar.update(1)
|
| 540 |
+
|
| 541 |
+
pbar.close()
|
| 542 |
+
|
| 543 |
+
if hasattr(lm, "_model"):
|
| 544 |
+
del lm._model
|
| 545 |
+
torch.cuda.empty_cache()
|
| 546 |
+
|
| 547 |
+
if WORLD_SIZE > 1:
|
| 548 |
+
# if multigpu, then gather data across all ranks to rank 0
|
| 549 |
+
# first gather logged samples across all ranks
|
| 550 |
+
for task_output in eval_tasks:
|
| 551 |
+
if log_samples:
|
| 552 |
+
# for task_name, task_samples in list(samples.items()):
|
| 553 |
+
full_samples = [None] * WORLD_SIZE if RANK == 0 else None
|
| 554 |
+
per_rank_samples = []
|
| 555 |
+
for sample in task_output.logged_samples:
|
| 556 |
+
per_rank_samples.append(sample)
|
| 557 |
+
|
| 558 |
+
torch.distributed.gather_object(
|
| 559 |
+
obj=per_rank_samples,
|
| 560 |
+
object_gather_list=full_samples,
|
| 561 |
+
dst=0,
|
| 562 |
+
)
|
| 563 |
+
|
| 564 |
+
if RANK == 0:
|
| 565 |
+
task_output.logged_samples = list(itertools.chain.from_iterable(full_samples))
|
| 566 |
+
|
| 567 |
+
# then collect metrics across all ranks
|
| 568 |
+
for metrics in task_output.sample_metrics:
|
| 569 |
+
metric_list = [None] * WORLD_SIZE if RANK == 0 else None
|
| 570 |
+
torch.distributed.gather_object(
|
| 571 |
+
obj=task_output.sample_metrics[metrics],
|
| 572 |
+
object_gather_list=metric_list,
|
| 573 |
+
dst=0,
|
| 574 |
+
)
|
| 575 |
+
if RANK == 0:
|
| 576 |
+
task_output.sample_metrics[metrics] = list(itertools.chain.from_iterable(metric_list))
|
| 577 |
+
|
| 578 |
+
dist.barrier() # Ensure all processes are synced before proceeding
|
| 579 |
+
|
| 580 |
+
if RANK == 0:
|
| 581 |
+
### Aggregate results over all datapoints ###
|
| 582 |
+
# aggregate results ; run bootstrap CIs
|
| 583 |
+
for task_output in eval_tasks:
|
| 584 |
+
task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters)
|
| 585 |
+
(
|
| 586 |
+
results,
|
| 587 |
+
samples,
|
| 588 |
+
configs,
|
| 589 |
+
versions,
|
| 590 |
+
num_fewshot,
|
| 591 |
+
higher_is_better,
|
| 592 |
+
) = consolidate_results(eval_tasks)
|
| 593 |
+
|
| 594 |
+
### Calculate group metrics ###
|
| 595 |
+
if bool(results):
|
| 596 |
+
results, versions, show_group_table, *_ = consolidate_group_results(results, versions, task_dict)
|
| 597 |
+
|
| 598 |
+
results_agg, group_agg = prepare_print_tasks(task_dict, results)
|
| 599 |
+
subtask_list = get_subtask_list(task_dict)
|
| 600 |
+
|
| 601 |
+
# collect all higher_is_better values for metrics
|
| 602 |
+
# in the group's subtasks.
|
| 603 |
+
# TODO: clean this up ; unify with the below metric_list loop?
|
| 604 |
+
_higher_is_better = {}
|
| 605 |
+
for group, task_list in subtask_list.items():
|
| 606 |
+
if len(task_list) != 0: # subtask list will list "task_name": [] for solo tasks
|
| 607 |
+
for task in task_list:
|
| 608 |
+
for m, h in higher_is_better[task].items():
|
| 609 |
+
if m not in _higher_is_better.keys():
|
| 610 |
+
_higher_is_better[m] = h
|
| 611 |
+
|
| 612 |
+
if m in _higher_is_better and _higher_is_better[m] is not None and _higher_is_better[m] != h:
|
| 613 |
+
eval_logger.warning(f"Higher_is_better values for metric {m} in group {group} are not consistent. Defaulting to None.")
|
| 614 |
+
_higher_is_better[m] = None
|
| 615 |
+
higher_is_better[group] = _higher_is_better
|
| 616 |
+
|
| 617 |
+
results_dict = {
|
| 618 |
+
"results": dict(results_agg.items()),
|
| 619 |
+
**({"groups": dict(group_agg.items())} if (bool(group_agg) & show_group_table) else {}),
|
| 620 |
+
"group_subtasks": dict(reversed(subtask_list.items())),
|
| 621 |
+
"configs": dict(sorted(configs.items())),
|
| 622 |
+
"versions": dict(sorted(versions.items())),
|
| 623 |
+
"n-shot": dict(sorted(num_fewshot.items())),
|
| 624 |
+
"higher_is_better": dict(sorted(higher_is_better.items())),
|
| 625 |
+
"n-samples": {
|
| 626 |
+
task_output.task_name: {
|
| 627 |
+
"original": len(task_output.task.eval_docs),
|
| 628 |
+
"effective": min(
|
| 629 |
+
limit if limit else len(task_output.task.eval_docs),
|
| 630 |
+
len(task_output.task.eval_docs),
|
| 631 |
+
),
|
| 632 |
+
}
|
| 633 |
+
for task_output in eval_tasks
|
| 634 |
+
},
|
| 635 |
+
}
|
| 636 |
+
if log_samples:
|
| 637 |
+
results_dict["samples"] = dict(samples)
|
| 638 |
+
else:
|
| 639 |
+
results_dict = None
|
| 640 |
+
|
| 641 |
+
if hasattr(lm, "accelerator"):
|
| 642 |
+
lm.accelerator.wait_for_everyone()
|
| 643 |
+
|
| 644 |
+
return results_dict
|
| 645 |
+
|
| 646 |
+
|
| 647 |
+
def request_caching_arg_to_dict(cache_requests: str) -> dict:
|
| 648 |
+
request_caching_args = {
|
| 649 |
+
"cache_requests": cache_requests in {"true", "refresh"},
|
| 650 |
+
"rewrite_requests_cache": cache_requests == "refresh",
|
| 651 |
+
"delete_requests_cache": cache_requests == "delete",
|
| 652 |
+
}
|
| 653 |
+
|
| 654 |
+
return request_caching_args
|
LaViDa/eval/lmms_eval/evaluator_utils.py
ADDED
|
@@ -0,0 +1,488 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# credit to https://github.com/EleutherAI/lm-evaluation-harness
|
| 2 |
+
import collections
|
| 3 |
+
import inspect
|
| 4 |
+
import math
|
| 5 |
+
import pathlib
|
| 6 |
+
import sys
|
| 7 |
+
from typing import List, Optional, Tuple, Union
|
| 8 |
+
|
| 9 |
+
from lmms_eval.api.group import ConfigurableGroup
|
| 10 |
+
from lmms_eval.api.metrics import (
|
| 11 |
+
aggregate_subtask_metrics,
|
| 12 |
+
pooled_sample_stderr,
|
| 13 |
+
stderr_for_metric,
|
| 14 |
+
)
|
| 15 |
+
from lmms_eval.api.task import Task
|
| 16 |
+
from lmms_eval.utils import eval_logger, positional_deprecated
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
class TaskOutput:
|
| 20 |
+
"""
|
| 21 |
+
Wrapper class for Task outputs.It contains various attributes and methods to manage and calculate metrics for the task.
|
| 22 |
+
|
| 23 |
+
Attributes:
|
| 24 |
+
task (object): The task object.
|
| 25 |
+
task_name (str): The name of the task.
|
| 26 |
+
task_config (dict): The configuration of the task.
|
| 27 |
+
version (str): The version of the task.
|
| 28 |
+
group_name (str): The name of the task group.
|
| 29 |
+
n_shot (int): The number of shots for the task.
|
| 30 |
+
task_alias (str): The alias of the task.
|
| 31 |
+
group_alias (str): The alias of the task group.
|
| 32 |
+
is_group (bool): Indicates if the task is a group.
|
| 33 |
+
logged_samples (list): The list of logged samples.
|
| 34 |
+
sample_len (int): The length of the samples.
|
| 35 |
+
sample_metrics (defaultdict): The dictionary of samples' metrics.
|
| 36 |
+
agg_metrics (defaultdict): The dictionary of aggregate metrics.
|
| 37 |
+
|
| 38 |
+
Methods:
|
| 39 |
+
from_taskdict(cls, task_name: str, task):
|
| 40 |
+
Creates a TaskOutput instance from a task dictionary.
|
| 41 |
+
|
| 42 |
+
calculate_aggregate_metric(bootstrap_iters=100000) -> None:
|
| 43 |
+
Calculates the aggregate metrics for the task.
|
| 44 |
+
"""
|
| 45 |
+
|
| 46 |
+
def __init__(
|
| 47 |
+
self,
|
| 48 |
+
task=None,
|
| 49 |
+
task_name=None,
|
| 50 |
+
task_config=None,
|
| 51 |
+
version=None,
|
| 52 |
+
group_name=None,
|
| 53 |
+
n_shot=None,
|
| 54 |
+
task_alias=None,
|
| 55 |
+
group_alias=None,
|
| 56 |
+
is_group=None,
|
| 57 |
+
):
|
| 58 |
+
self.task = task
|
| 59 |
+
self.task_config = task_config
|
| 60 |
+
self.task_name = task_name
|
| 61 |
+
self.group_name = group_name
|
| 62 |
+
self.version = version
|
| 63 |
+
self.n_shot = n_shot
|
| 64 |
+
self.task_alias = task_alias
|
| 65 |
+
self.group_alias = group_alias
|
| 66 |
+
self.is_group = is_group
|
| 67 |
+
self.logged_samples = []
|
| 68 |
+
self.sample_len = None
|
| 69 |
+
self.sample_metrics = collections.defaultdict(list)
|
| 70 |
+
self.agg_metrics = collections.defaultdict(list)
|
| 71 |
+
self.args = None
|
| 72 |
+
|
| 73 |
+
@classmethod
|
| 74 |
+
def from_taskdict(cls, task_name: str, task):
|
| 75 |
+
if isinstance(task, tuple):
|
| 76 |
+
group_name, task = task
|
| 77 |
+
else:
|
| 78 |
+
group_name = None
|
| 79 |
+
if not task:
|
| 80 |
+
# these gets filtered out in get_task_list
|
| 81 |
+
# once they are added to group hierarchy
|
| 82 |
+
is_group = True
|
| 83 |
+
return cls(task=task, task_name=task_name, is_group=is_group, group_name=group_name)
|
| 84 |
+
version = task.VERSION
|
| 85 |
+
task_config = dict(task.dump_config())
|
| 86 |
+
if (n_shot := task_config.get("num_fewshot")) == 0:
|
| 87 |
+
meta_config = task_config.get("metadata", {})
|
| 88 |
+
if isinstance(meta_config, dict):
|
| 89 |
+
n_shot = meta_config.get("num_fewshot", 0)
|
| 90 |
+
else:
|
| 91 |
+
eval_logger.info(f"No metadata found in task config for {task_name}, using default n_shot=0")
|
| 92 |
+
n_shot = 0
|
| 93 |
+
task_alias = task_config.get("alias")
|
| 94 |
+
group_alias = task_config.get("group_alias")
|
| 95 |
+
return cls(
|
| 96 |
+
task=task,
|
| 97 |
+
task_name=task_name,
|
| 98 |
+
task_config=task_config,
|
| 99 |
+
group_name=group_name,
|
| 100 |
+
version=version,
|
| 101 |
+
n_shot=n_shot,
|
| 102 |
+
task_alias=task_alias,
|
| 103 |
+
group_alias=group_alias,
|
| 104 |
+
)
|
| 105 |
+
|
| 106 |
+
def calculate_aggregate_metric(self, bootstrap_iters=100000) -> None:
|
| 107 |
+
for (metric, filter_key), items in self.sample_metrics.items():
|
| 108 |
+
if metric in self.task.aggregation():
|
| 109 |
+
agg_fn = self.task.aggregation()[metric]
|
| 110 |
+
metric_key = f"{metric},{filter_key}"
|
| 111 |
+
if "args" in inspect.signature(agg_fn).parameters:
|
| 112 |
+
self.agg_metrics[metric_key] = agg_fn(items, args=self.task.args)
|
| 113 |
+
else:
|
| 114 |
+
self.agg_metrics[metric_key] = agg_fn(items)
|
| 115 |
+
self.sample_len = len(items) # TODO: same sample size for each metric?
|
| 116 |
+
if isinstance(bootstrap_iters, int):
|
| 117 |
+
stderr_fn = stderr_for_metric(
|
| 118 |
+
metric=agg_fn,
|
| 119 |
+
bootstrap_iters=min(bootstrap_iters, 100) if metric in ["bleu", "chrf", "ter"] else bootstrap_iters,
|
| 120 |
+
)
|
| 121 |
+
self.agg_metrics[f"{metric}_stderr,{filter_key}"] = stderr_fn(items) if (stderr_fn and len(items) > 1) else "N/A"
|
| 122 |
+
else:
|
| 123 |
+
raise ValueError(f"Received bootstrap_iters '{bootstrap_iters}' but expected an integer. Set to 0 to turn off stderr calculations.")
|
| 124 |
+
|
| 125 |
+
def __repr__(self):
|
| 126 |
+
return f"TaskOutput(task_name={self.task_name}, " f"group_name={self.group_name}, " f"version={self.version}, " f"n_shot={self.n_shot}, " f"task_alias={self.task_alias}, " f"group_alias={self.group_alias})"
|
| 127 |
+
|
| 128 |
+
|
| 129 |
+
def get_task_list(task_dict: dict) -> List[TaskOutput]:
|
| 130 |
+
outputs = []
|
| 131 |
+
for task_name, task_obj in task_dict.items():
|
| 132 |
+
if isinstance(task_obj, dict):
|
| 133 |
+
_outputs = get_task_list(task_obj)
|
| 134 |
+
outputs.extend(_outputs)
|
| 135 |
+
else:
|
| 136 |
+
task_output = TaskOutput.from_taskdict(task_name, task_obj)
|
| 137 |
+
outputs.append(task_output)
|
| 138 |
+
|
| 139 |
+
return outputs
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def get_subtask_list(task_dict, task_root=None, depth=0):
|
| 143 |
+
subtask_list = {}
|
| 144 |
+
for group_obj, task_obj in task_dict.items():
|
| 145 |
+
if isinstance(group_obj, ConfigurableGroup):
|
| 146 |
+
# group_name = group_obj.group_name
|
| 147 |
+
group_name = group_obj.group_name
|
| 148 |
+
else:
|
| 149 |
+
group_name = group_obj
|
| 150 |
+
if isinstance(task_obj, dict):
|
| 151 |
+
_subtask_list = get_subtask_list(task_obj, task_root=group_name, depth=depth + 1)
|
| 152 |
+
if task_root:
|
| 153 |
+
subtask_list.setdefault((task_root, depth), []).extend([_task for (_task, _depth) in _subtask_list.keys() if (_depth - 1) == depth])
|
| 154 |
+
|
| 155 |
+
subtask_list = {**subtask_list, **_subtask_list}
|
| 156 |
+
else:
|
| 157 |
+
if isinstance(task_obj, ConfigurableGroup):
|
| 158 |
+
# group_or_task_name = task_obj.group_name
|
| 159 |
+
group_or_task_name = task_obj.group_name
|
| 160 |
+
elif isinstance(task_obj, Task):
|
| 161 |
+
# group_or_task_name = task_obj.task_name
|
| 162 |
+
group_or_task_name = task_obj.task_name
|
| 163 |
+
|
| 164 |
+
if task_root is None:
|
| 165 |
+
subtask_list.setdefault((group_or_task_name, depth), [])
|
| 166 |
+
else:
|
| 167 |
+
subtask_list.setdefault((task_root, depth), []).append(group_or_task_name)
|
| 168 |
+
|
| 169 |
+
if depth == 0:
|
| 170 |
+
_subtask_list = {}
|
| 171 |
+
for group_key, task_list in subtask_list.items():
|
| 172 |
+
group_name, depth = group_key
|
| 173 |
+
_subtask_list[group_name] = task_list
|
| 174 |
+
subtask_list = _subtask_list
|
| 175 |
+
|
| 176 |
+
return subtask_list
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
def print_writeout(task) -> None:
|
| 180 |
+
for inst in task.instances:
|
| 181 |
+
# print the prompt for the first few documents
|
| 182 |
+
if inst.doc_id < 1:
|
| 183 |
+
eval_logger.info(
|
| 184 |
+
f"Task: {task}; document {inst.doc_id}; context prompt (starting on next line):\
|
| 185 |
+
\n{inst.args[0]}\n(end of prompt on previous line)\ntarget string or answer choice index (starting on next line):\n{task.doc_to_target(inst.doc)}\n(end of target on previous line)"
|
| 186 |
+
)
|
| 187 |
+
eval_logger.info(f"Request: {str(inst)}")
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
def get_sample_size(task, limit: Optional[int]) -> Union[int, None]:
|
| 191 |
+
if limit is not None:
|
| 192 |
+
limit = int(math.ceil(len(task.eval_docs) * limit)) if limit < 1.0 else int(limit)
|
| 193 |
+
return limit
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def prepare_print_tasks(
|
| 197 |
+
task_dict: dict,
|
| 198 |
+
results: dict,
|
| 199 |
+
task_depth=0,
|
| 200 |
+
group_depth=0,
|
| 201 |
+
) -> Tuple[dict, dict]:
|
| 202 |
+
"""
|
| 203 |
+
@param task_dict: Dictionary representing the group hierarchy of tasks. Each key is a group name and its
|
| 204 |
+
value is a list of task names.
|
| 205 |
+
@param results: Dictionary containing the results of each task. Each key is a
|
| 206 |
+
group name and its value is a dictionary of task results.
|
| 207 |
+
@param task_depth: The indentation level for printing the task
|
| 208 |
+
hierarchy. Default is 0.
|
| 209 |
+
@param group_depth: The indentation level for printing the group
|
| 210 |
+
hierarchy. Default is 0.
|
| 211 |
+
@return: A tuple of two dictionaries: results_agg and groups_agg. results_agg contains
|
| 212 |
+
aggregated results for each task, and groups_agg contains aggregated results for each group.
|
| 213 |
+
|
| 214 |
+
Prepares the task hierarchy and aggregates the results for each task and group recursively for printing.
|
| 215 |
+
"""
|
| 216 |
+
|
| 217 |
+
def _sort_task_dict(task_dict):
|
| 218 |
+
"""
|
| 219 |
+
Helper utility. Sorts the task dict at the current level of the hierarchy based on alphabetized task name.
|
| 220 |
+
Required so that we end up sorting within each sub-header correctly.
|
| 221 |
+
"""
|
| 222 |
+
|
| 223 |
+
return dict(
|
| 224 |
+
sorted(
|
| 225 |
+
task_dict.items(),
|
| 226 |
+
key=lambda item: item[0].group_name if isinstance(item[0], ConfigurableGroup) else item[0],
|
| 227 |
+
)
|
| 228 |
+
)
|
| 229 |
+
|
| 230 |
+
task_agg = collections.defaultdict(dict)
|
| 231 |
+
group_agg = collections.defaultdict(dict)
|
| 232 |
+
task_dict = _sort_task_dict(task_dict)
|
| 233 |
+
for task_or_group_name, task_or_group_obj in task_dict.items():
|
| 234 |
+
tab_string = " " * task_depth + "- " if task_depth > 0 else ""
|
| 235 |
+
if isinstance(task_or_group_name, ConfigurableGroup):
|
| 236 |
+
# string_name = task_or_group_name.group_name
|
| 237 |
+
name = task_or_group_name.group_name
|
| 238 |
+
from_configurable_group = True
|
| 239 |
+
task_or_group_obj = _sort_task_dict(task_or_group_obj)
|
| 240 |
+
elif isinstance(task_or_group_name, str):
|
| 241 |
+
name = task_or_group_name
|
| 242 |
+
if isinstance(task_or_group_obj, Task):
|
| 243 |
+
# string_name = task_or_group_obj.task_name
|
| 244 |
+
name = task_or_group_obj.task_name
|
| 245 |
+
from_configurable_group = False
|
| 246 |
+
|
| 247 |
+
task_agg[name] = results[name].copy()
|
| 248 |
+
if from_configurable_group:
|
| 249 |
+
if task_or_group_name.group_alias is not None:
|
| 250 |
+
alias = task_or_group_name.group_alias
|
| 251 |
+
else:
|
| 252 |
+
alias = task_or_group_name.group
|
| 253 |
+
else:
|
| 254 |
+
if "alias" in task_agg[name]:
|
| 255 |
+
alias = task_agg[name]["alias"]
|
| 256 |
+
else:
|
| 257 |
+
alias = name
|
| 258 |
+
|
| 259 |
+
task_agg[name]["alias"] = tab_string + alias
|
| 260 |
+
if "samples" in task_agg[name]:
|
| 261 |
+
task_agg[name].pop("samples")
|
| 262 |
+
|
| 263 |
+
if from_configurable_group and (" " not in results[name]):
|
| 264 |
+
group_tab_string = " " * group_depth + "- " if group_depth > 0 else ""
|
| 265 |
+
group_agg[name] = results[name].copy()
|
| 266 |
+
group_agg[name]["alias"] = group_tab_string + alias
|
| 267 |
+
if "samples" in group_agg[name]:
|
| 268 |
+
group_agg[name].pop("samples")
|
| 269 |
+
|
| 270 |
+
if isinstance(task_or_group_obj, dict):
|
| 271 |
+
task_depth += 1
|
| 272 |
+
group_depth += 1
|
| 273 |
+
_task_agg, _group_agg = prepare_print_tasks(task_or_group_obj, results, task_depth, group_depth)
|
| 274 |
+
task_agg = {
|
| 275 |
+
**task_agg,
|
| 276 |
+
**_task_agg,
|
| 277 |
+
}
|
| 278 |
+
group_agg = {**group_agg, **_group_agg}
|
| 279 |
+
task_depth -= 1
|
| 280 |
+
group_depth -= 1
|
| 281 |
+
return task_agg, group_agg
|
| 282 |
+
|
| 283 |
+
|
| 284 |
+
def consolidate_results(
|
| 285 |
+
eval_tasks: List[TaskOutput],
|
| 286 |
+
) -> Tuple[dict, dict, dict, dict, dict, dict]:
|
| 287 |
+
"""
|
| 288 |
+
@param eval_tasks: list(TaskOutput).
|
| 289 |
+
@return: A tuple containing the consolidated results, samples, configs, versions, and num_fewshot.
|
| 290 |
+
|
| 291 |
+
Consolidates the results of multiple evaluation tasks into a single structure.
|
| 292 |
+
|
| 293 |
+
The method iterates over each evaluation instance and extracts relevant information to create the consolidated
|
| 294 |
+
results structure. The consolidated results structure has the following properties:
|
| 295 |
+
|
| 296 |
+
- results: A defaultdict with task names as keys and dictionaries as values. Each dictionary contains
|
| 297 |
+
metric/filter pairs as keys and corresponding metric values as values. The "alias" key is used to store task
|
| 298 |
+
aliases specified in the task configuration.
|
| 299 |
+
- samples: A defaultdict with task names as keys and lists of log samples as values.
|
| 300 |
+
- configs: A defaultdict with task names as keys and task configurations as values.
|
| 301 |
+
- versions: A defaultdict with task names as keys and task versions as values.
|
| 302 |
+
- num_fewshot: A defaultdict with task names as keys and number of few-shot samples as values.
|
| 303 |
+
- higher_is_better: A defaultdict with task names as keys and indicators of whether higher values are better
|
| 304 |
+
for each metric as values.
|
| 305 |
+
|
| 306 |
+
The method then returns the consolidated results, samples, configs, versions, and num_fewshot as a tuple.
|
| 307 |
+
"""
|
| 308 |
+
# stores the final result for each task, for each metric/filter pair.
|
| 309 |
+
results = collections.defaultdict(dict)
|
| 310 |
+
# logs info about each document evaluated.
|
| 311 |
+
samples = collections.defaultdict(list)
|
| 312 |
+
# store num-fewshot value per task
|
| 313 |
+
num_fewshot = collections.defaultdict(int)
|
| 314 |
+
# Tracks the YAML configs of all chosen task
|
| 315 |
+
configs = collections.defaultdict(dict)
|
| 316 |
+
# Tracks each task's version.
|
| 317 |
+
versions = collections.defaultdict(dict)
|
| 318 |
+
# Track `higher_is_better` for each metric
|
| 319 |
+
higher_is_better = collections.defaultdict(dict)
|
| 320 |
+
|
| 321 |
+
for task_output in eval_tasks:
|
| 322 |
+
if "task_alias" in (task_config := task_output.task_config):
|
| 323 |
+
results[task_output.task_name]["alias"] = task_config["task_alias"]
|
| 324 |
+
else:
|
| 325 |
+
results[task_output.task_name]["alias"] = task_output.task_name
|
| 326 |
+
if group_alias := task_output.group_alias:
|
| 327 |
+
if group_alias not in results and (group_name := task_output.group_name):
|
| 328 |
+
results[group_name]["alias"] = group_alias
|
| 329 |
+
num_fewshot[task_output.task_name] = task_output.n_shot
|
| 330 |
+
configs[task_output.task_name] = task_output.task_config
|
| 331 |
+
versions[task_output.task_name] = task_output.version
|
| 332 |
+
samples[task_output.task_name] = task_output.logged_samples
|
| 333 |
+
higher_is_better[task_output.task_name] = task_output.task.higher_is_better()
|
| 334 |
+
for (metric, filter_key), items in task_output.sample_metrics.items():
|
| 335 |
+
metric_key = f"{metric},{filter_key}"
|
| 336 |
+
results[task_output.task_name][metric_key] = task_output.agg_metrics[metric_key]
|
| 337 |
+
results[task_output.task_name]["samples"] = task_output.sample_len
|
| 338 |
+
results[task_output.task_name][f"{metric}_stderr,{filter_key}"] = task_output.agg_metrics[f"{metric}_stderr,{filter_key}"]
|
| 339 |
+
return results, samples, configs, versions, num_fewshot, higher_is_better
|
| 340 |
+
|
| 341 |
+
|
| 342 |
+
def consolidate_group_results(
|
| 343 |
+
results,
|
| 344 |
+
versions,
|
| 345 |
+
task_dict,
|
| 346 |
+
task_root=None,
|
| 347 |
+
show_group_table=False,
|
| 348 |
+
task_aggregation_list=None,
|
| 349 |
+
) -> Tuple[dict, dict, bool, Union[None, dict]]:
|
| 350 |
+
"""
|
| 351 |
+
(Recursively) calculates groups' aggregated metrics and updates the results and versions dictionaries with this info.
|
| 352 |
+
|
| 353 |
+
@return: a tuple [results, versions, show_group_table, task_aggregation_list] with formats described below:
|
| 354 |
+
|
| 355 |
+
- results: A defaultdict with task names (and, after this function is called, group names of
|
| 356 |
+
groups that perform aggregation) as keys, and dictionaries with "alias" and metric,filter_name pairs as keys.
|
| 357 |
+
- versions: A defaultdict with task names (and, after this function is called, group names of
|
| 358 |
+
groups that perform aggregation) as keys, and float values representing the task or group's version if a version is specified. (defaulting to None).
|
| 359 |
+
- show_group_table: a boolean which is true if there exists a group that requires printing of its aggregated scores in a group table.
|
| 360 |
+
- task_aggregation_list: a defaultdict listing the subtasks to average over to produce a given group's end metric.
|
| 361 |
+
|
| 362 |
+
The method then returns the updated results, versions, show_group_table, and task_aggregation_list as a tuple.
|
| 363 |
+
In the top-level invocation of this function, task_aggregation_list is ignored.
|
| 364 |
+
"""
|
| 365 |
+
if task_root is None:
|
| 366 |
+
task_root = {}
|
| 367 |
+
|
| 368 |
+
if task_aggregation_list is None:
|
| 369 |
+
task_aggregation_list = {}
|
| 370 |
+
|
| 371 |
+
for group_or_task, group_or_task_info in task_dict.items():
|
| 372 |
+
# Convert to string
|
| 373 |
+
if isinstance(group_or_task, ConfigurableGroup):
|
| 374 |
+
group_config = group_or_task.config
|
| 375 |
+
group_or_task = group_or_task.group_name
|
| 376 |
+
else:
|
| 377 |
+
group_config = None
|
| 378 |
+
|
| 379 |
+
if isinstance(group_or_task_info, Task):
|
| 380 |
+
if task_root:
|
| 381 |
+
task_aggregation_list.setdefault(task_root, []).append(group_or_task_info.task_name)
|
| 382 |
+
else:
|
| 383 |
+
(
|
| 384 |
+
results,
|
| 385 |
+
versions,
|
| 386 |
+
show_group_table,
|
| 387 |
+
_task_aggregation_list,
|
| 388 |
+
) = consolidate_group_results(
|
| 389 |
+
results,
|
| 390 |
+
versions,
|
| 391 |
+
group_or_task_info,
|
| 392 |
+
group_or_task,
|
| 393 |
+
show_group_table,
|
| 394 |
+
task_aggregation_list,
|
| 395 |
+
)
|
| 396 |
+
if task_root:
|
| 397 |
+
task_aggregation_list.setdefault(task_root, []).extend(task_aggregation_list.get(group_or_task, []))
|
| 398 |
+
|
| 399 |
+
if (group_config is None) or (group_config["aggregate_metric_list"] is None):
|
| 400 |
+
results[group_or_task][" "] = " "
|
| 401 |
+
continue
|
| 402 |
+
|
| 403 |
+
if "aggregate_metric_list" in group_config:
|
| 404 |
+
agg_metric_list = group_config["aggregate_metric_list"]
|
| 405 |
+
|
| 406 |
+
show_group_table = show_group_table | bool(group_config["aggregate_metric_list"])
|
| 407 |
+
|
| 408 |
+
task_list = _task_aggregation_list[group_or_task]
|
| 409 |
+
|
| 410 |
+
metric_list = list({key for task in task_list for key in results[task].keys() if "_stderr" not in key and key not in ["task", "alias", "samples"]})
|
| 411 |
+
for metric in metric_list:
|
| 412 |
+
stderr = "_stderr,".join(metric.split(","))
|
| 413 |
+
|
| 414 |
+
# gather metrics, sizes, and stderrs from subtasks
|
| 415 |
+
metrics = [results[task][metric] for task in task_list if metric in results[task]] # TODO: copy?
|
| 416 |
+
stderrs = [results[task][stderr] for task in task_list if stderr in results[task]]
|
| 417 |
+
sizes = [results[task]["samples"] for task in task_list if metric in results[task]]
|
| 418 |
+
|
| 419 |
+
for metric_config in agg_metric_list:
|
| 420 |
+
for filter_name in metric_config["filter_list"]:
|
| 421 |
+
# if metric != ",".join([metric_config["metric"], filter_name]):
|
| 422 |
+
# continue
|
| 423 |
+
if metric_config["metric"] not in metric:
|
| 424 |
+
continue
|
| 425 |
+
|
| 426 |
+
# compute group's pooled metric and stderr
|
| 427 |
+
if metric_config["aggregation"] == "mean":
|
| 428 |
+
aggregate_fn = aggregate_subtask_metrics
|
| 429 |
+
elif callable(metric_config["aggregation"]):
|
| 430 |
+
aggregate_fn = metric_config["aggregation"]
|
| 431 |
+
else:
|
| 432 |
+
raise ValueError(f"Currently, only 'mean' is supported for automatically aggregating scores across groups' subtasks. Got '{metric_config['aggregation']}' for group '{group_or_task}'")
|
| 433 |
+
|
| 434 |
+
results[group_or_task][metric] = aggregate_fn(
|
| 435 |
+
metrics,
|
| 436 |
+
sizes,
|
| 437 |
+
metric_config["weight_by_size"],
|
| 438 |
+
)
|
| 439 |
+
# TODO: calculate groups' metrics using arbitrary agg fns
|
| 440 |
+
if "N/A" in stderrs:
|
| 441 |
+
results[group_or_task][stderr] = "N/A"
|
| 442 |
+
else:
|
| 443 |
+
# NOTE: this assumes we are using the mean to aggregate. There are warnings about this elsewhere
|
| 444 |
+
results[group_or_task][stderr] = pooled_sample_stderr(stderrs, sizes)
|
| 445 |
+
|
| 446 |
+
results[group_or_task]["samples"] = sum(sizes)
|
| 447 |
+
group_metadata = group_config.get("metadata", None)
|
| 448 |
+
if group_metadata is not None:
|
| 449 |
+
versions[group_or_task] = group_metadata.get("version", None)
|
| 450 |
+
# print(results)
|
| 451 |
+
return results, versions, show_group_table, task_aggregation_list
|
| 452 |
+
|
| 453 |
+
|
| 454 |
+
@positional_deprecated
|
| 455 |
+
def find_test_root(start_path: pathlib.Path) -> pathlib.Path:
|
| 456 |
+
"""
|
| 457 |
+
Search upward in the directory tree to a maximum of three layers
|
| 458 |
+
to find and return the package root (containing the 'tests' folder)
|
| 459 |
+
"""
|
| 460 |
+
cur_path = start_path.resolve()
|
| 461 |
+
max_layers = 3
|
| 462 |
+
for _ in range(max_layers):
|
| 463 |
+
if (cur_path / "tests" / "test_version_stable.py").exists():
|
| 464 |
+
return cur_path
|
| 465 |
+
else:
|
| 466 |
+
cur_path = cur_path.parent.resolve()
|
| 467 |
+
raise FileNotFoundError(f"Unable to find package root within {max_layers} upwards" + f"of {start_path}")
|
| 468 |
+
|
| 469 |
+
|
| 470 |
+
@positional_deprecated
|
| 471 |
+
def run_task_tests(task_list: List[str]):
|
| 472 |
+
"""
|
| 473 |
+
Find the package root and run the tests for the given tasks
|
| 474 |
+
"""
|
| 475 |
+
import pytest
|
| 476 |
+
|
| 477 |
+
package_root = find_test_root(start_path=pathlib.Path(__file__))
|
| 478 |
+
task_string = " or ".join(task_list)
|
| 479 |
+
args = [
|
| 480 |
+
f"{package_root}/tests/test_version_stable.py",
|
| 481 |
+
f"--rootdir={package_root}",
|
| 482 |
+
"-k",
|
| 483 |
+
f"{task_string}",
|
| 484 |
+
]
|
| 485 |
+
sys.path.append(str(package_root))
|
| 486 |
+
pytest_return_val = pytest.main(args)
|
| 487 |
+
if pytest_return_val:
|
| 488 |
+
raise ValueError(f"Not all tests for the specified tasks ({task_list}) ran successfully! Error code: {pytest_return_val}")
|
LaViDa/eval/lmms_eval/filters/__init__.py
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from lmms_eval.api.filter import Filter, FilterEnsemble
|
| 2 |
+
|
| 3 |
+
from . import extraction, selection, transformation
|
| 4 |
+
|
| 5 |
+
FILTER_REGISTRY = {
|
| 6 |
+
"take_first": selection.TakeFirstFilter,
|
| 7 |
+
"regex": extraction.RegexFilter,
|
| 8 |
+
"majority_vote": selection.MajorityVoteFilter,
|
| 9 |
+
"take_first_k": selection.TakeKFilter,
|
| 10 |
+
"remove_whitespace": extraction.WhitespaceFilter,
|
| 11 |
+
"lowercase": transformation.LowercaseFilter,
|
| 12 |
+
"uppercase": transformation.UppercaseFilter,
|
| 13 |
+
"map": transformation.MapFilter,
|
| 14 |
+
"multi_choice_regex": extraction.MultiChoiceRegexFilter,
|
| 15 |
+
# TODO: implement this filter. either it should take in an arbitrary "scoring"/reward function
|
| 16 |
+
# that takes an input and returns a scalar and then should select the max reward,
|
| 17 |
+
# or should implement different filters for different ways of handling a reward model's inference.
|
| 18 |
+
# "arg_max": selection.ArgMaxFilter,
|
| 19 |
+
}
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def get_filter(filter_name):
|
| 23 |
+
if filter_name in FILTER_REGISTRY:
|
| 24 |
+
return FILTER_REGISTRY[filter_name]
|
| 25 |
+
else:
|
| 26 |
+
return filter_name
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def build_filter_ensemble(filter_name, components):
|
| 30 |
+
"""
|
| 31 |
+
Create a filtering pipeline.
|
| 32 |
+
"""
|
| 33 |
+
filters = []
|
| 34 |
+
for function, kwargs in components:
|
| 35 |
+
if kwargs is None:
|
| 36 |
+
f = get_filter(function)()
|
| 37 |
+
else:
|
| 38 |
+
# create a filter given its name in the registry
|
| 39 |
+
f = get_filter(function)(**kwargs) # TODO: pass kwargs to filters properly
|
| 40 |
+
# add the filter as a pipeline step
|
| 41 |
+
filters.append(f)
|
| 42 |
+
|
| 43 |
+
return FilterEnsemble(name=filter_name, filters=filters)
|
LaViDa/eval/lmms_eval/filters/decontamination.py
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from lmms_eval.api.filter import Filter
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
class DecontaminationFilter(Filter):
|
| 5 |
+
"""
|
| 6 |
+
A filter which evaluates
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
name = "track_decontamination"
|
| 10 |
+
|
| 11 |
+
def __init__(self, path) -> None:
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
TODO: make sure only ever run one time on the train set (should this be cached as a class var? keyed by value for "path").
|
| 15 |
+
should further cache result on a given (task_name, doc_id)
|
| 16 |
+
"""
|
| 17 |
+
self._decontam_results = None
|
| 18 |
+
|
| 19 |
+
def apply(self, resps, docs) -> None:
|
| 20 |
+
"""
|
| 21 |
+
Return {"no_contamination", "only_contamination"} keys for the 2 different subsets
|
| 22 |
+
"""
|
| 23 |
+
pass
|
LaViDa/eval/lmms_eval/filters/extraction.py
ADDED
|
@@ -0,0 +1,282 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
import re
|
| 3 |
+
import sys
|
| 4 |
+
import unicodedata
|
| 5 |
+
|
| 6 |
+
import openai
|
| 7 |
+
|
| 8 |
+
from lmms_eval.api.filter import Filter
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
class WhitespaceFilter(Filter):
|
| 12 |
+
""" """
|
| 13 |
+
|
| 14 |
+
def __init__(self) -> None:
|
| 15 |
+
pass
|
| 16 |
+
|
| 17 |
+
def apply(self, resps, docs):
|
| 18 |
+
def filter_set(inst):
|
| 19 |
+
filtered_resp = []
|
| 20 |
+
for resp in inst:
|
| 21 |
+
if resp.startswith(" "):
|
| 22 |
+
resp = resp[1:]
|
| 23 |
+
|
| 24 |
+
filtered_resp.append(resp)
|
| 25 |
+
|
| 26 |
+
return filtered_resp
|
| 27 |
+
|
| 28 |
+
filtered_resps = [filter_set(resp) for resp in resps]
|
| 29 |
+
|
| 30 |
+
return filtered_resps
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
class RegexFilter(Filter):
|
| 34 |
+
""" """
|
| 35 |
+
|
| 36 |
+
def __init__(
|
| 37 |
+
self,
|
| 38 |
+
regex_pattern: str = r"#### (\-?[0-9\.\,]+)",
|
| 39 |
+
group_select=0,
|
| 40 |
+
fallback: str = "[invalid]",
|
| 41 |
+
) -> None:
|
| 42 |
+
"""
|
| 43 |
+
pass a string `regex` to run `re.compile(r"regex")` on.
|
| 44 |
+
`fallback` defines the output returned if no matches for the regex are located.
|
| 45 |
+
"""
|
| 46 |
+
self.regex_pattern = regex_pattern
|
| 47 |
+
self.regex = re.compile(regex_pattern)
|
| 48 |
+
self.group_select = group_select
|
| 49 |
+
self.fallback = fallback
|
| 50 |
+
|
| 51 |
+
def apply(self, resps, docs):
|
| 52 |
+
# here, we assume we have a list, in which each element is
|
| 53 |
+
# a list of model responses for some particular input/target pair.
|
| 54 |
+
# so we process each of these (same input/target response sets)
|
| 55 |
+
# independently (and keep them a list.)
|
| 56 |
+
def filter_set(inst):
|
| 57 |
+
filtered = []
|
| 58 |
+
for resp in inst:
|
| 59 |
+
match = self.regex.findall(resp)
|
| 60 |
+
if match:
|
| 61 |
+
match = match[self.group_select]
|
| 62 |
+
if isinstance(match, tuple):
|
| 63 |
+
match = [m for m in match if m][0]
|
| 64 |
+
match = match.strip()
|
| 65 |
+
else:
|
| 66 |
+
match = self.fallback
|
| 67 |
+
filtered.append(match)
|
| 68 |
+
return filtered
|
| 69 |
+
|
| 70 |
+
# print(resps)
|
| 71 |
+
filtered_resps = list(map(lambda x: filter_set(x), resps))
|
| 72 |
+
# print(filtered_resps)
|
| 73 |
+
|
| 74 |
+
return filtered_resps
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
class MultiChoiceRegexFilter(RegexFilter):
|
| 78 |
+
"""
|
| 79 |
+
A filter used to extract a model's answer on multiple choice questions with
|
| 80 |
+
letter answers. assumes each document has a "choices" field
|
| 81 |
+
containing the list of answer choices and that the answer label symbols
|
| 82 |
+
are of the form (A), (B), (C), ... or A, B, C.
|
| 83 |
+
"""
|
| 84 |
+
|
| 85 |
+
def __init__(
|
| 86 |
+
self,
|
| 87 |
+
regex_pattern: str = r"#### (\-?[0-9\.\,]+)",
|
| 88 |
+
group_select=0,
|
| 89 |
+
fallback: str = "[invalid]",
|
| 90 |
+
ignore_case=False,
|
| 91 |
+
ignore_punctuation=False,
|
| 92 |
+
regexes_to_ignore=None,
|
| 93 |
+
) -> None:
|
| 94 |
+
"""
|
| 95 |
+
regex_pattern: The basic regex pattern to use. If fails to match, we will use the customized match procedure
|
| 96 |
+
- step 1 : We parse the choices between ([A-Z])s then try to find these choices in the response.
|
| 97 |
+
- step 2 : We parse the choice with regex :[\s]*([A-?]), where ? varies by number of choices.
|
| 98 |
+
group_select: Selects the (group_select)th match from the findall result.
|
| 99 |
+
ignore_case: Ignores the case during step 1 matching
|
| 100 |
+
ignore_punctuation: Remove the punctuation during step 1 matching
|
| 101 |
+
regexes_to_ignore: Remove these regexes during step 1 matching
|
| 102 |
+
"""
|
| 103 |
+
super().__init__(regex_pattern, group_select, fallback)
|
| 104 |
+
self.ignore_case = ignore_case
|
| 105 |
+
self.ignore_punctuation = ignore_punctuation
|
| 106 |
+
self.regexes_to_ignore = regexes_to_ignore
|
| 107 |
+
|
| 108 |
+
def apply(self, resps, docs):
|
| 109 |
+
# here, we assume we have a list, in which each element is
|
| 110 |
+
# a list of model responses for some particular input/target pair.
|
| 111 |
+
# so we process each of these (same input/target response sets)
|
| 112 |
+
# independently (and keep them a list.)
|
| 113 |
+
|
| 114 |
+
def find_match(regex, resp, convert_dict={}):
|
| 115 |
+
match = regex.findall(resp)
|
| 116 |
+
if match:
|
| 117 |
+
match = match[self.group_select]
|
| 118 |
+
if isinstance(match, tuple):
|
| 119 |
+
match = [m for m in match if m][0]
|
| 120 |
+
match = match.strip()
|
| 121 |
+
if match and match in convert_dict:
|
| 122 |
+
match = convert_dict[match]
|
| 123 |
+
return match
|
| 124 |
+
|
| 125 |
+
punct_tbl = dict.fromkeys(i for i in range(sys.maxunicode) if unicodedata.category(chr(i)).startswith("P"))
|
| 126 |
+
|
| 127 |
+
def filter_ignores(st):
|
| 128 |
+
if self.regexes_to_ignore is not None:
|
| 129 |
+
for s in self.regexes_to_ignore:
|
| 130 |
+
st = re.sub(s, "", st)
|
| 131 |
+
|
| 132 |
+
if self.ignore_case:
|
| 133 |
+
st = st.lower()
|
| 134 |
+
|
| 135 |
+
if self.ignore_punctuation:
|
| 136 |
+
# https://stackoverflow.com/a/266162
|
| 137 |
+
st = st.translate(punct_tbl)
|
| 138 |
+
return st
|
| 139 |
+
|
| 140 |
+
filtered_resps = []
|
| 141 |
+
|
| 142 |
+
for r, doc in zip(resps, docs):
|
| 143 |
+
fallback_regexes = []
|
| 144 |
+
choice_to_alpha = {}
|
| 145 |
+
next_alpha = "A"
|
| 146 |
+
|
| 147 |
+
without_paren_fallback_regexes = []
|
| 148 |
+
without_paren_to_target = {}
|
| 149 |
+
|
| 150 |
+
choices = doc["choices"]
|
| 151 |
+
for c in choices:
|
| 152 |
+
m = filter_ignores(c.strip())
|
| 153 |
+
fallback_regexes.append(f"{re.escape(m)}")
|
| 154 |
+
choice_to_alpha[m] = f"({next_alpha})"
|
| 155 |
+
|
| 156 |
+
without_paren_fallback_regexes.append(next_alpha)
|
| 157 |
+
without_paren_to_target[next_alpha] = f"({next_alpha})"
|
| 158 |
+
|
| 159 |
+
next_alpha = chr(ord(next_alpha) + 1)
|
| 160 |
+
fallback_regex = re.compile("|".join(fallback_regexes))
|
| 161 |
+
without_paren_fallback_regex = "|".join(without_paren_fallback_regexes)
|
| 162 |
+
without_paren_fallback_regex = re.compile(f":[\s]*({without_paren_fallback_regex})")
|
| 163 |
+
|
| 164 |
+
filtered = []
|
| 165 |
+
for resp in r:
|
| 166 |
+
match = find_match(self.regex, resp)
|
| 167 |
+
if not match:
|
| 168 |
+
match = find_match(fallback_regex, filter_ignores(resp), choice_to_alpha)
|
| 169 |
+
if not match:
|
| 170 |
+
match = find_match(without_paren_fallback_regex, resp, without_paren_to_target)
|
| 171 |
+
if not match:
|
| 172 |
+
match = self.fallback
|
| 173 |
+
filtered.append(match)
|
| 174 |
+
filtered_resps.append(filtered)
|
| 175 |
+
|
| 176 |
+
return filtered_resps
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
class ExtendedRegexFilter(RegexFilter):
|
| 180 |
+
punct_tbl = dict.fromkeys(i for i in range(sys.maxunicode) if unicodedata.category(chr(i)).startswith("P"))
|
| 181 |
+
|
| 182 |
+
def __init__(
|
| 183 |
+
self,
|
| 184 |
+
regex_pattern: str = r"#### (\-?[0-9\.\,]+)",
|
| 185 |
+
group_select=0,
|
| 186 |
+
fallback: str = "[invalid]",
|
| 187 |
+
ignore_case=False,
|
| 188 |
+
ignore_punctuation=False,
|
| 189 |
+
regexes_to_ignore=None,
|
| 190 |
+
) -> None:
|
| 191 |
+
super().__init__(regex_pattern, group_select, fallback)
|
| 192 |
+
self.ignore_case = ignore_case
|
| 193 |
+
self.ignore_punctuation = ignore_punctuation
|
| 194 |
+
self.regexes_to_ignore = regexes_to_ignore
|
| 195 |
+
|
| 196 |
+
def filter_ignores(self, st):
|
| 197 |
+
if self.regexes_to_ignore is not None:
|
| 198 |
+
for s in self.regexes_to_ignore:
|
| 199 |
+
st = re.sub(s, "", st)
|
| 200 |
+
|
| 201 |
+
if self.ignore_case:
|
| 202 |
+
st = st.lower()
|
| 203 |
+
|
| 204 |
+
if self.ignore_punctuation:
|
| 205 |
+
# https://stackoverflow.com/a/266162
|
| 206 |
+
st = st.translate(self.punct_tbl)
|
| 207 |
+
return st
|
| 208 |
+
|
| 209 |
+
def find_match(self, regex, resp, convert_dict={}):
|
| 210 |
+
match = regex.findall(resp)
|
| 211 |
+
if match:
|
| 212 |
+
match = match[self.group_select]
|
| 213 |
+
if isinstance(match, tuple):
|
| 214 |
+
match = [m for m in match if m][0]
|
| 215 |
+
match = match.strip()
|
| 216 |
+
if match and match in convert_dict:
|
| 217 |
+
match = convert_dict[match]
|
| 218 |
+
return match
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
# Designed for the AI2D/RealworldQA dataset
|
| 222 |
+
class SimpleMultiChoiceRegexFilter(ExtendedRegexFilter):
|
| 223 |
+
def __init__(self, *args, **kwargs):
|
| 224 |
+
"""
|
| 225 |
+
regex_pattern: The basic regex pattern to use. If fails to match, we will use the customized match procedure
|
| 226 |
+
- step 1 : We parse the choices between ([A-Z])s then try to find these choices in the response.
|
| 227 |
+
- step 2 : We parse the choice with regex :[\s]*([A-?]), where ? varies by number of choices.
|
| 228 |
+
group_select: Selects the (group_select)th match from the findall result.
|
| 229 |
+
ignore_case: Ignores the case during step 1 matching
|
| 230 |
+
ignore_punctuation: Remove the punctuation during step 1 matching
|
| 231 |
+
regexes_to_ignore: Remove these regexes during step 1 matching
|
| 232 |
+
"""
|
| 233 |
+
super().__init__(*args, **kwargs)
|
| 234 |
+
|
| 235 |
+
def apply(self, resps, docs):
|
| 236 |
+
# here, we assume we have a list, in which each element is
|
| 237 |
+
# a list of model responses for some particular input/target pair.
|
| 238 |
+
# so we process each of these (same input/target response sets)
|
| 239 |
+
# independently (and keep them a list.)
|
| 240 |
+
|
| 241 |
+
filtered_resps = []
|
| 242 |
+
|
| 243 |
+
for r, doc in zip(resps, docs):
|
| 244 |
+
fallback_regexes = []
|
| 245 |
+
choice_to_alpha = {}
|
| 246 |
+
next_alpha = "A"
|
| 247 |
+
|
| 248 |
+
without_paren_fallback_regexes = []
|
| 249 |
+
without_paren_to_target = {}
|
| 250 |
+
|
| 251 |
+
# Regex to extract multiple choice options from the question
|
| 252 |
+
multiple_choices_regex = re.compile(r"\b([A-Z])\.\s+([^\n]*)")
|
| 253 |
+
matches = multiple_choices_regex.findall(doc["question"])
|
| 254 |
+
|
| 255 |
+
# Build regex patterns and mappings for each choice
|
| 256 |
+
for m in matches:
|
| 257 |
+
choice_text = m[1].strip()
|
| 258 |
+
fallback_regexes.append(f"{re.escape(choice_text)}")
|
| 259 |
+
choice_to_alpha[choice_text] = next_alpha
|
| 260 |
+
|
| 261 |
+
next_alpha = chr(ord(next_alpha) + 1)
|
| 262 |
+
|
| 263 |
+
# Compile regex to match any of the extracted choices
|
| 264 |
+
fallback_regex = re.compile("|".join(fallback_regexes))
|
| 265 |
+
|
| 266 |
+
# Process each response
|
| 267 |
+
filtered = []
|
| 268 |
+
for resp in r:
|
| 269 |
+
# Remove any punctuation and extra spaces
|
| 270 |
+
cleaned_resp = re.sub(r"[^\w\s]", "", resp).strip()
|
| 271 |
+
# Try to match cleaned response with the choice text
|
| 272 |
+
match = fallback_regex.search(cleaned_resp)
|
| 273 |
+
if match and match.group() in choice_to_alpha:
|
| 274 |
+
# Map the matched choice text back to its corresponding letter
|
| 275 |
+
filtered.append(choice_to_alpha[match.group()])
|
| 276 |
+
else:
|
| 277 |
+
# If no match, return the cleaned response
|
| 278 |
+
filtered.append(cleaned_resp)
|
| 279 |
+
|
| 280 |
+
filtered_resps.append(filtered[0])
|
| 281 |
+
|
| 282 |
+
return filtered_resps
|
LaViDa/eval/lmms_eval/filters/selection.py
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from collections import Counter
|
| 2 |
+
|
| 3 |
+
from lmms_eval.api.filter import Filter
|
| 4 |
+
from lmms_eval.api.registry import register_filter
|
| 5 |
+
|
| 6 |
+
# TODO: implement "arg_max" filter. either it should take in an arbitrary "scoring"/reward function
|
| 7 |
+
# that takes an input and returns a scalar and then should select the max reward,
|
| 8 |
+
# or should implement different filters for different ways of handling a reward model's inference.
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
@register_filter("take_first")
|
| 12 |
+
class TakeFirstFilter(Filter):
|
| 13 |
+
def __init__(self) -> None:
|
| 14 |
+
"""
|
| 15 |
+
Can define custom behavior here, if an individual instantiation of a Filter class should have state.
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
def apply(self, resps, docs):
|
| 19 |
+
"""
|
| 20 |
+
Assuming each entry of `resps` is a list of model responses, we discard all but the first response.
|
| 21 |
+
"""
|
| 22 |
+
return map(lambda r: r[0], resps)
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
@register_filter("take_first_k")
|
| 26 |
+
class TakeKFilter(Filter):
|
| 27 |
+
def __init__(self, **kwargs) -> None:
|
| 28 |
+
self.k = kwargs.pop("k")
|
| 29 |
+
|
| 30 |
+
super().__init__(**kwargs)
|
| 31 |
+
|
| 32 |
+
def apply(self, resps, docs):
|
| 33 |
+
# need resp to be subscriptable to check below
|
| 34 |
+
resps = list(resps)
|
| 35 |
+
# check we have at least k responses per doc, else we can't take the first k
|
| 36 |
+
assert len(resps[0]) >= self.k, f"Need at least {self.k} responses per doc to take first {self.k}, but got {len(resps[0])} only! Please increase TaskConfig.repeats ."
|
| 37 |
+
selected = map(lambda r: r[: self.k], resps)
|
| 38 |
+
return selected
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
@register_filter("majority_vote")
|
| 42 |
+
class MajorityVoteFilter(Filter):
|
| 43 |
+
def __init__(self) -> None:
|
| 44 |
+
"""
|
| 45 |
+
Can define custom behavior here, if an individual instantiation of a Filter class should have state.
|
| 46 |
+
"""
|
| 47 |
+
|
| 48 |
+
def apply(self, resps, docs):
|
| 49 |
+
"""
|
| 50 |
+
Each entry of `resps` is a list of model responses.
|
| 51 |
+
We select the response that occurs most frequently in each entry of `resps`.
|
| 52 |
+
"""
|
| 53 |
+
|
| 54 |
+
def select_majority(resp):
|
| 55 |
+
counts = Counter(resp)
|
| 56 |
+
vote = counts.most_common(1)[0][0]
|
| 57 |
+
return vote
|
| 58 |
+
|
| 59 |
+
return map(lambda r: [select_majority(r)], resps)
|
LaViDa/eval/lmms_eval/filters/transformation.py
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from lmms_eval.api.filter import Filter
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
class LowercaseFilter(Filter):
|
| 5 |
+
def __init__(self) -> None:
|
| 6 |
+
pass
|
| 7 |
+
|
| 8 |
+
def apply(self, resps, docs):
|
| 9 |
+
def filter_set(inst):
|
| 10 |
+
return [resp.lower() for resp in inst]
|
| 11 |
+
|
| 12 |
+
return [filter_set(resp) for resp in resps]
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
class UppercaseFilter(Filter):
|
| 16 |
+
def __init__(self) -> None:
|
| 17 |
+
pass
|
| 18 |
+
|
| 19 |
+
def apply(self, resps, docs):
|
| 20 |
+
def filter_set(inst):
|
| 21 |
+
return [resp.upper() for resp in inst]
|
| 22 |
+
|
| 23 |
+
return [filter_set(resp) for resp in resps]
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
class MapFilter(Filter):
|
| 27 |
+
def __init__(self, mapping_dict: dict = {}, default_value=None) -> None:
|
| 28 |
+
"""
|
| 29 |
+
Initializes the MapFilter with a given mapping dictionary and default value.
|
| 30 |
+
|
| 31 |
+
Args:
|
| 32 |
+
- mapping_dict (dict): A dictionary containing the key-value mappings.
|
| 33 |
+
Default is an empty dictionary.
|
| 34 |
+
- default_value (Any): The value to be returned when a key is not found in the mapping_dict.
|
| 35 |
+
Default is None.
|
| 36 |
+
|
| 37 |
+
Example:
|
| 38 |
+
mapper = MapFilter({'A': 1, 'B': 2}, default_value=0)
|
| 39 |
+
"""
|
| 40 |
+
assert isinstance(mapping_dict, dict), "Provided mapping_dict is not a dictionary"
|
| 41 |
+
self.mapping_dict = mapping_dict
|
| 42 |
+
self.default_value = default_value
|
| 43 |
+
|
| 44 |
+
def apply(self, resps, docs):
|
| 45 |
+
def filter_set(inst):
|
| 46 |
+
return [self.mapping_dict.get(resp, self.default_value) for resp in inst]
|
| 47 |
+
|
| 48 |
+
return [filter_set(resp) for resp in resps]
|
LaViDa/eval/lmms_eval/loggers/__init__.py
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from .evaluation_tracker import EvaluationTracker
|
| 2 |
+
from .wandb_logger import WandbLogger
|
LaViDa/eval/lmms_eval/loggers/evaluation_tracker.py
ADDED
|
@@ -0,0 +1,452 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# code is adapted from https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/loggers/evaluation_tracker.py
|
| 2 |
+
import json
|
| 3 |
+
import os
|
| 4 |
+
import re
|
| 5 |
+
import time
|
| 6 |
+
from collections import defaultdict
|
| 7 |
+
from dataclasses import asdict, dataclass
|
| 8 |
+
from datetime import datetime
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
|
| 11 |
+
from datasets import load_dataset
|
| 12 |
+
from datasets.utils.metadata import MetadataConfigs
|
| 13 |
+
from huggingface_hub import DatasetCard, DatasetCardData, HfApi, hf_hub_url
|
| 14 |
+
from huggingface_hub.utils import build_hf_headers, get_session, hf_raise_for_status
|
| 15 |
+
|
| 16 |
+
from lmms_eval.utils import (
|
| 17 |
+
eval_logger,
|
| 18 |
+
get_datetime_str,
|
| 19 |
+
get_file_datetime,
|
| 20 |
+
get_file_task_name,
|
| 21 |
+
get_results_filenames,
|
| 22 |
+
get_sample_results_filenames,
|
| 23 |
+
handle_non_serializable,
|
| 24 |
+
hash_string,
|
| 25 |
+
sanitize_list,
|
| 26 |
+
sanitize_model_name,
|
| 27 |
+
sanitize_task_name,
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
@dataclass(init=False)
|
| 32 |
+
class GeneralConfigTracker:
|
| 33 |
+
"""
|
| 34 |
+
Tracker for the evaluation parameters.
|
| 35 |
+
|
| 36 |
+
Attributes:
|
| 37 |
+
model_source (str): Source of the model (e.g. Hugging Face, GGUF, etc.)
|
| 38 |
+
model_name (str): Name of the model.
|
| 39 |
+
model_name_sanitized (str): Sanitized model name for directory creation.
|
| 40 |
+
start_time (float): Start time of the experiment. Logged at class init.
|
| 41 |
+
end_time (float): Start time of the experiment. Logged when calling [`GeneralConfigTracker.log_end_time`]
|
| 42 |
+
total_evaluation_time_seconds (str): Inferred total evaluation time in seconds (from the start and end times).
|
| 43 |
+
"""
|
| 44 |
+
|
| 45 |
+
model_source: str = None
|
| 46 |
+
model_name: str = None
|
| 47 |
+
model_name_sanitized: str = None
|
| 48 |
+
system_instruction: str = None
|
| 49 |
+
system_instruction_sha: str = None
|
| 50 |
+
fewshot_as_multiturn: bool = None
|
| 51 |
+
chat_template: str = None
|
| 52 |
+
chat_template_sha: str = None
|
| 53 |
+
start_time: float = None
|
| 54 |
+
end_time: float = None
|
| 55 |
+
total_evaluation_time_seconds: str = None
|
| 56 |
+
|
| 57 |
+
def __init__(self) -> None:
|
| 58 |
+
"""Starts the evaluation timer."""
|
| 59 |
+
self.start_time = time.perf_counter()
|
| 60 |
+
|
| 61 |
+
@staticmethod
|
| 62 |
+
def _get_model_name(model_args: str) -> str:
|
| 63 |
+
"""Extracts the model name from the model arguments."""
|
| 64 |
+
|
| 65 |
+
def extract_model_name(model_args: str, key: str) -> str:
|
| 66 |
+
"""Extracts the model name from the model arguments using a key."""
|
| 67 |
+
args_after_key = model_args.split(key)[1]
|
| 68 |
+
return args_after_key.split(",")[0]
|
| 69 |
+
|
| 70 |
+
# order does matter, e.g. peft and delta are provided together with pretrained
|
| 71 |
+
prefixes = ["peft=", "delta=", "pretrained=", "model=", "path=", "engine="]
|
| 72 |
+
for prefix in prefixes:
|
| 73 |
+
if prefix in model_args:
|
| 74 |
+
return extract_model_name(model_args, prefix)
|
| 75 |
+
return ""
|
| 76 |
+
|
| 77 |
+
def log_experiment_args(
|
| 78 |
+
self,
|
| 79 |
+
model_source: str,
|
| 80 |
+
model_args: str,
|
| 81 |
+
system_instruction: str,
|
| 82 |
+
chat_template: str,
|
| 83 |
+
fewshot_as_multiturn: bool,
|
| 84 |
+
) -> None:
|
| 85 |
+
"""Logs model parameters and job ID."""
|
| 86 |
+
self.model_source = model_source
|
| 87 |
+
self.model_name = GeneralConfigTracker._get_model_name(model_args)
|
| 88 |
+
self.model_name_sanitized = sanitize_model_name(self.model_name)
|
| 89 |
+
self.system_instruction = system_instruction
|
| 90 |
+
self.system_instruction_sha = hash_string(system_instruction) if system_instruction else None
|
| 91 |
+
self.chat_template = chat_template
|
| 92 |
+
self.chat_template_sha = hash_string(chat_template) if chat_template else None
|
| 93 |
+
self.fewshot_as_multiturn = fewshot_as_multiturn
|
| 94 |
+
|
| 95 |
+
def log_end_time(self) -> None:
|
| 96 |
+
"""Logs the end time of the evaluation and calculates the total evaluation time."""
|
| 97 |
+
self.end_time = time.perf_counter()
|
| 98 |
+
self.total_evaluation_time_seconds = str(self.end_time - self.start_time)
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
class EvaluationTracker:
|
| 102 |
+
"""
|
| 103 |
+
Keeps track and saves relevant information of the evaluation process.
|
| 104 |
+
Compiles the data from trackers and writes it to files, which can be published to the Hugging Face hub if requested.
|
| 105 |
+
"""
|
| 106 |
+
|
| 107 |
+
def __init__(
|
| 108 |
+
self,
|
| 109 |
+
output_path: str = None,
|
| 110 |
+
hub_results_org: str = "",
|
| 111 |
+
hub_repo_name: str = "",
|
| 112 |
+
details_repo_name: str = "",
|
| 113 |
+
results_repo_name: str = "",
|
| 114 |
+
push_results_to_hub: bool = False,
|
| 115 |
+
push_samples_to_hub: bool = False,
|
| 116 |
+
public_repo: bool = False,
|
| 117 |
+
token: str = "",
|
| 118 |
+
leaderboard_url: str = "",
|
| 119 |
+
point_of_contact: str = "",
|
| 120 |
+
gated: bool = False,
|
| 121 |
+
) -> None:
|
| 122 |
+
"""
|
| 123 |
+
Creates all the necessary loggers for evaluation tracking.
|
| 124 |
+
|
| 125 |
+
Args:
|
| 126 |
+
output_path (str): Path to save the results. If not provided, the results won't be saved.
|
| 127 |
+
hub_results_org (str): The Hugging Face organization to push the results to. If not provided, the results will be pushed to the owner of the Hugging Face token.
|
| 128 |
+
hub_repo_name (str): The name of the Hugging Face repository to push the results to. If not provided, the results will be pushed to `lm-eval-results`.
|
| 129 |
+
details_repo_name (str): The name of the Hugging Face repository to push the details to. If not provided, the results will be pushed to `lm-eval-results`.
|
| 130 |
+
result_repo_name (str): The name of the Hugging Face repository to push the results to. If not provided, the results will not be pushed and will be found in the details_hub_repo.
|
| 131 |
+
push_results_to_hub (bool): Whether to push the results to the Hugging Face hub.
|
| 132 |
+
push_samples_to_hub (bool): Whether to push the samples to the Hugging Face hub.
|
| 133 |
+
public_repo (bool): Whether to push the results to a public or private repository.
|
| 134 |
+
token (str): Token to use when pushing to the Hugging Face hub. This token should have write access to `hub_results_org`.
|
| 135 |
+
leaderboard_url (str): URL to the leaderboard on the Hugging Face hub on the dataset card.
|
| 136 |
+
point_of_contact (str): Contact information on the Hugging Face hub dataset card.
|
| 137 |
+
gated (bool): Whether to gate the repository.
|
| 138 |
+
"""
|
| 139 |
+
self.general_config_tracker = GeneralConfigTracker()
|
| 140 |
+
|
| 141 |
+
self.output_path = output_path
|
| 142 |
+
self.push_results_to_hub = push_results_to_hub
|
| 143 |
+
self.push_samples_to_hub = push_samples_to_hub
|
| 144 |
+
self.public_repo = public_repo
|
| 145 |
+
self.leaderboard_url = leaderboard_url
|
| 146 |
+
self.point_of_contact = point_of_contact
|
| 147 |
+
self.api = HfApi(token=token) if token else None
|
| 148 |
+
self.gated_repo = gated
|
| 149 |
+
|
| 150 |
+
if not self.api and (push_results_to_hub or push_samples_to_hub):
|
| 151 |
+
raise ValueError("Hugging Face token is not defined, but 'push_results_to_hub' or 'push_samples_to_hub' is set to True. " "Please provide a valid Hugging Face token by setting the HF_TOKEN environment variable.")
|
| 152 |
+
|
| 153 |
+
if self.api and hub_results_org == "" and (push_results_to_hub or push_samples_to_hub):
|
| 154 |
+
hub_results_org = self.api.whoami()["name"]
|
| 155 |
+
eval_logger.warning(f"hub_results_org was not specified. Results will be pushed to '{hub_results_org}'.")
|
| 156 |
+
|
| 157 |
+
if hub_repo_name == "":
|
| 158 |
+
details_repo_name = details_repo_name if details_repo_name != "" else "lmms-eval-results"
|
| 159 |
+
results_repo_name = results_repo_name if results_repo_name != "" else details_repo_name
|
| 160 |
+
else:
|
| 161 |
+
details_repo_name = hub_repo_name
|
| 162 |
+
results_repo_name = hub_repo_name
|
| 163 |
+
eval_logger.warning("hub_repo_name was specified. Both details and results will be pushed to the same repository. Using hub_repo_name is no longer recommended, details_repo_name and results_repo_name should be used instead.")
|
| 164 |
+
|
| 165 |
+
self.details_repo = f"{hub_results_org}/{details_repo_name}"
|
| 166 |
+
self.details_repo_private = f"{hub_results_org}/{details_repo_name}-private"
|
| 167 |
+
self.results_repo = f"{hub_results_org}/{results_repo_name}"
|
| 168 |
+
self.results_repo_private = f"{hub_results_org}/{results_repo_name}-private"
|
| 169 |
+
|
| 170 |
+
def save_results_aggregated(
|
| 171 |
+
self,
|
| 172 |
+
results: dict,
|
| 173 |
+
samples: dict,
|
| 174 |
+
datetime_str: str,
|
| 175 |
+
) -> None:
|
| 176 |
+
"""
|
| 177 |
+
Saves the aggregated results and samples to the output path and pushes them to the Hugging Face hub if requested.
|
| 178 |
+
|
| 179 |
+
Args:
|
| 180 |
+
results (dict): The aggregated results to save.
|
| 181 |
+
samples (dict): The samples results to save.
|
| 182 |
+
datetime_str (str): The datetime string to use for the results file.
|
| 183 |
+
"""
|
| 184 |
+
self.general_config_tracker.log_end_time()
|
| 185 |
+
|
| 186 |
+
if self.output_path:
|
| 187 |
+
try:
|
| 188 |
+
eval_logger.info("Saving results aggregated")
|
| 189 |
+
|
| 190 |
+
# calculate cumulative hash for each task - only if samples are provided
|
| 191 |
+
task_hashes = {}
|
| 192 |
+
if samples:
|
| 193 |
+
for task_name, task_samples in samples.items():
|
| 194 |
+
sample_hashes = [s["doc_hash"] + s["prompt_hash"] + s["target_hash"] for s in task_samples]
|
| 195 |
+
task_hashes[task_name] = hash_string("".join(sample_hashes))
|
| 196 |
+
|
| 197 |
+
# update initial results dict
|
| 198 |
+
results.update({"task_hashes": task_hashes})
|
| 199 |
+
results.update(asdict(self.general_config_tracker))
|
| 200 |
+
dumped = json.dumps(
|
| 201 |
+
results,
|
| 202 |
+
indent=2,
|
| 203 |
+
default=handle_non_serializable,
|
| 204 |
+
ensure_ascii=False,
|
| 205 |
+
)
|
| 206 |
+
|
| 207 |
+
path = Path(self.output_path if self.output_path else Path.cwd())
|
| 208 |
+
path = path.joinpath(self.general_config_tracker.model_name_sanitized)
|
| 209 |
+
path.mkdir(parents=True, exist_ok=True)
|
| 210 |
+
|
| 211 |
+
self.date_id = datetime_str.replace(":", "-")
|
| 212 |
+
file_results_aggregated = path.joinpath(f"{self.date_id}_results.json")
|
| 213 |
+
file_results_aggregated.open("w", encoding="utf-8").write(dumped)
|
| 214 |
+
|
| 215 |
+
if self.api and self.push_results_to_hub:
|
| 216 |
+
repo_id = self.results_repo if self.public_repo else self.results_repo_private
|
| 217 |
+
self.api.create_repo(
|
| 218 |
+
repo_id=repo_id,
|
| 219 |
+
repo_type="dataset",
|
| 220 |
+
private=not self.public_repo,
|
| 221 |
+
exist_ok=True,
|
| 222 |
+
)
|
| 223 |
+
self.api.upload_file(
|
| 224 |
+
repo_id=repo_id,
|
| 225 |
+
path_or_fileobj=str(path.joinpath(f"{self.date_id}_results.json")),
|
| 226 |
+
path_in_repo=os.path.join(
|
| 227 |
+
self.general_config_tracker.model_name,
|
| 228 |
+
f"{self.date_id}_results.json",
|
| 229 |
+
),
|
| 230 |
+
repo_type="dataset",
|
| 231 |
+
commit_message=f"Adding aggregated results for {self.general_config_tracker.model_name}",
|
| 232 |
+
)
|
| 233 |
+
eval_logger.info("Successfully pushed aggregated results to the Hugging Face Hub. " f"You can find them at: {repo_id}")
|
| 234 |
+
|
| 235 |
+
except Exception as e:
|
| 236 |
+
eval_logger.warning("Could not save results aggregated")
|
| 237 |
+
eval_logger.info(repr(e))
|
| 238 |
+
else:
|
| 239 |
+
eval_logger.info("Output path not provided, skipping saving results aggregated")
|
| 240 |
+
|
| 241 |
+
def save_results_samples(
|
| 242 |
+
self,
|
| 243 |
+
task_name: str,
|
| 244 |
+
samples: dict,
|
| 245 |
+
) -> None:
|
| 246 |
+
"""
|
| 247 |
+
Saves the samples results to the output path and pushes them to the Hugging Face hub if requested.
|
| 248 |
+
|
| 249 |
+
Args:
|
| 250 |
+
task_name (str): The task name to save the samples for.
|
| 251 |
+
samples (dict): The samples results to save.
|
| 252 |
+
"""
|
| 253 |
+
if self.output_path:
|
| 254 |
+
try:
|
| 255 |
+
eval_logger.info(f"Saving per-sample results for: {task_name}")
|
| 256 |
+
|
| 257 |
+
path = Path(self.output_path if self.output_path else Path.cwd())
|
| 258 |
+
path = path.joinpath(self.general_config_tracker.model_name_sanitized)
|
| 259 |
+
path.mkdir(parents=True, exist_ok=True)
|
| 260 |
+
|
| 261 |
+
file_results_samples = path.joinpath(f"{self.date_id}_samples_{task_name}.jsonl")
|
| 262 |
+
|
| 263 |
+
for sample in samples:
|
| 264 |
+
# we first need to sanitize arguments and resps
|
| 265 |
+
# otherwise we won't be able to load the dataset
|
| 266 |
+
# using the datasets library
|
| 267 |
+
arguments = {}
|
| 268 |
+
for key, value in enumerate(sample["arguments"][1]): # update metadata into args
|
| 269 |
+
arguments[key] = value
|
| 270 |
+
|
| 271 |
+
sample["input"] = sample["arguments"][0]
|
| 272 |
+
sample["resps"] = sanitize_list(sample["resps"])
|
| 273 |
+
sample["filtered_resps"] = sanitize_list(sample["filtered_resps"])
|
| 274 |
+
sample["arguments"] = arguments
|
| 275 |
+
sample["target"] = str(sample["target"])
|
| 276 |
+
|
| 277 |
+
sample_dump = (
|
| 278 |
+
json.dumps(
|
| 279 |
+
sample,
|
| 280 |
+
default=handle_non_serializable,
|
| 281 |
+
ensure_ascii=False,
|
| 282 |
+
)
|
| 283 |
+
+ "\n"
|
| 284 |
+
)
|
| 285 |
+
|
| 286 |
+
with open(file_results_samples, "a", encoding="utf-8") as f:
|
| 287 |
+
f.write(sample_dump)
|
| 288 |
+
|
| 289 |
+
if self.api and self.push_samples_to_hub:
|
| 290 |
+
repo_id = self.details_repo if self.public_repo else self.details_repo_private
|
| 291 |
+
self.api.create_repo(
|
| 292 |
+
repo_id=repo_id,
|
| 293 |
+
repo_type="dataset",
|
| 294 |
+
private=not self.public_repo,
|
| 295 |
+
exist_ok=True,
|
| 296 |
+
)
|
| 297 |
+
try:
|
| 298 |
+
if self.gated_repo:
|
| 299 |
+
headers = build_hf_headers()
|
| 300 |
+
r = get_session().put(
|
| 301 |
+
url=f"https://huggingface.co/api/datasets/{repo_id}/settings",
|
| 302 |
+
headers=headers,
|
| 303 |
+
json={"gated": "auto"},
|
| 304 |
+
)
|
| 305 |
+
hf_raise_for_status(r)
|
| 306 |
+
except Exception as e:
|
| 307 |
+
eval_logger.warning("Could not gate the repository")
|
| 308 |
+
eval_logger.info(repr(e))
|
| 309 |
+
self.api.upload_folder(
|
| 310 |
+
repo_id=repo_id,
|
| 311 |
+
folder_path=str(path),
|
| 312 |
+
path_in_repo=self.general_config_tracker.model_name_sanitized,
|
| 313 |
+
repo_type="dataset",
|
| 314 |
+
commit_message=f"Adding samples results for {task_name} to {self.general_config_tracker.model_name}",
|
| 315 |
+
)
|
| 316 |
+
eval_logger.info(f"Successfully pushed sample results for task: {task_name} to the Hugging Face Hub. " f"You can find them at: {repo_id}")
|
| 317 |
+
|
| 318 |
+
except Exception as e:
|
| 319 |
+
eval_logger.warning("Could not save sample results")
|
| 320 |
+
eval_logger.info(repr(e))
|
| 321 |
+
else:
|
| 322 |
+
eval_logger.info("Output path not provided, skipping saving sample results")
|
| 323 |
+
|
| 324 |
+
def recreate_metadata_card(self) -> None:
|
| 325 |
+
"""
|
| 326 |
+
Creates a metadata card for the evaluation results dataset and pushes it to the Hugging Face hub.
|
| 327 |
+
"""
|
| 328 |
+
|
| 329 |
+
eval_logger.info("Recreating metadata card")
|
| 330 |
+
repo_id = self.details_repo if self.public_repo else self.details_repo_private
|
| 331 |
+
|
| 332 |
+
files_in_repo = self.api.list_repo_files(repo_id=repo_id, repo_type="dataset")
|
| 333 |
+
results_files = get_results_filenames(files_in_repo)
|
| 334 |
+
sample_files = get_sample_results_filenames(files_in_repo)
|
| 335 |
+
|
| 336 |
+
# Build a dictionary to store the latest evaluation datetime for:
|
| 337 |
+
# - Each tested model and its aggregated results
|
| 338 |
+
# - Each task and sample results, if existing
|
| 339 |
+
# i.e. {
|
| 340 |
+
# "org__model_name__gsm8k": "2021-09-01T12:00:00",
|
| 341 |
+
# "org__model_name__ifeval": "2021-09-01T12:00:00",
|
| 342 |
+
# "org__model_name__results": "2021-09-01T12:00:00"
|
| 343 |
+
# }
|
| 344 |
+
latest_task_results_datetime = defaultdict(lambda: datetime.min.isoformat())
|
| 345 |
+
|
| 346 |
+
for file_path in sample_files:
|
| 347 |
+
file_path = Path(file_path)
|
| 348 |
+
filename = file_path.name
|
| 349 |
+
model_name = file_path.parent
|
| 350 |
+
task_name = get_file_task_name(filename)
|
| 351 |
+
results_datetime = get_file_datetime(filename)
|
| 352 |
+
task_name_sanitized = sanitize_task_name(task_name)
|
| 353 |
+
# Results and sample results for the same model and task will have the same datetime
|
| 354 |
+
samples_key = f"{model_name}__{task_name_sanitized}"
|
| 355 |
+
results_key = f"{model_name}__results"
|
| 356 |
+
latest_datetime = max(
|
| 357 |
+
latest_task_results_datetime[samples_key],
|
| 358 |
+
results_datetime,
|
| 359 |
+
)
|
| 360 |
+
latest_task_results_datetime[samples_key] = latest_datetime
|
| 361 |
+
latest_task_results_datetime[results_key] = max(
|
| 362 |
+
latest_task_results_datetime[results_key],
|
| 363 |
+
latest_datetime,
|
| 364 |
+
)
|
| 365 |
+
|
| 366 |
+
# Create metadata card
|
| 367 |
+
card_metadata = MetadataConfigs()
|
| 368 |
+
|
| 369 |
+
# Add the latest aggregated results to the metadata card for easy access
|
| 370 |
+
for file_path in results_files:
|
| 371 |
+
file_path = Path(file_path)
|
| 372 |
+
results_filename = file_path.name
|
| 373 |
+
model_name = file_path.parent
|
| 374 |
+
eval_date = get_file_datetime(results_filename)
|
| 375 |
+
eval_date_sanitized = re.sub(r"[^\w\.]", "_", eval_date)
|
| 376 |
+
results_filename = Path("**") / Path(results_filename).name
|
| 377 |
+
config_name = f"{model_name}__results"
|
| 378 |
+
sanitized_last_eval_date_results = re.sub(r"[^\w\.]", "_", latest_task_results_datetime[config_name])
|
| 379 |
+
|
| 380 |
+
if eval_date_sanitized == sanitized_last_eval_date_results:
|
| 381 |
+
# Ensure that all results files are listed in the metadata card
|
| 382 |
+
current_results = card_metadata.get(config_name, {"data_files": []})
|
| 383 |
+
current_results["data_files"].append({"split": eval_date_sanitized, "path": [str(results_filename)]})
|
| 384 |
+
card_metadata[config_name] = current_results
|
| 385 |
+
# If the results file is the newest, update the "latest" field in the metadata card
|
| 386 |
+
card_metadata[config_name]["data_files"].append({"split": "latest", "path": [str(results_filename)]})
|
| 387 |
+
|
| 388 |
+
# Add the tasks details configs
|
| 389 |
+
for file_path in sample_files:
|
| 390 |
+
file_path = Path(file_path)
|
| 391 |
+
filename = file_path.name
|
| 392 |
+
model_name = file_path.parent
|
| 393 |
+
task_name = get_file_task_name(filename)
|
| 394 |
+
eval_date = get_file_datetime(filename)
|
| 395 |
+
task_name_sanitized = sanitize_task_name(task_name)
|
| 396 |
+
eval_date_sanitized = re.sub(r"[^\w\.]", "_", eval_date)
|
| 397 |
+
results_filename = Path("**") / Path(filename).name
|
| 398 |
+
config_name = f"{model_name}__{task_name_sanitized}"
|
| 399 |
+
sanitized_last_eval_date_results = re.sub(r"[^\w\.]", "_", latest_task_results_datetime[config_name])
|
| 400 |
+
if eval_date_sanitized == sanitized_last_eval_date_results:
|
| 401 |
+
# Ensure that all sample results files are listed in the metadata card
|
| 402 |
+
current_details_for_task = card_metadata.get(config_name, {"data_files": []})
|
| 403 |
+
current_details_for_task["data_files"].append({"split": eval_date_sanitized, "path": [str(results_filename)]})
|
| 404 |
+
card_metadata[config_name] = current_details_for_task
|
| 405 |
+
# If the samples results file is the newest, update the "latest" field in the metadata card
|
| 406 |
+
card_metadata[config_name]["data_files"].append({"split": "latest", "path": [str(results_filename)]})
|
| 407 |
+
|
| 408 |
+
# Get latest results and extract info to update metadata card examples
|
| 409 |
+
latest_datetime = max(latest_task_results_datetime.values())
|
| 410 |
+
latest_model_name = max(latest_task_results_datetime, key=lambda k: latest_task_results_datetime[k])
|
| 411 |
+
last_results_file = [f for f in results_files if latest_datetime.replace(":", "-") in f][0]
|
| 412 |
+
last_results_file_path = hf_hub_url(repo_id=repo_id, filename=last_results_file, repo_type="dataset")
|
| 413 |
+
latest_results_file = load_dataset("json", data_files=last_results_file_path, split="train")
|
| 414 |
+
results_dict = latest_results_file["results"][0]
|
| 415 |
+
new_dictionary = {"all": results_dict}
|
| 416 |
+
new_dictionary.update(results_dict)
|
| 417 |
+
results_string = json.dumps(new_dictionary, indent=4)
|
| 418 |
+
|
| 419 |
+
dataset_summary = "Dataset automatically created during the evaluation run of model "
|
| 420 |
+
if self.general_config_tracker.model_source == "hf":
|
| 421 |
+
dataset_summary += f"[{self.general_config_tracker.model_name}](https://huggingface.co/{self.general_config_tracker.model_name})\n"
|
| 422 |
+
else:
|
| 423 |
+
dataset_summary += f"{self.general_config_tracker.model_name}\n"
|
| 424 |
+
dataset_summary += (
|
| 425 |
+
f"The dataset is composed of {len(card_metadata)-1} configuration(s), each one corresponding to one of the evaluated task.\n\n"
|
| 426 |
+
f"The dataset has been created from {len(results_files)} run(s). Each run can be found as a specific split in each "
|
| 427 |
+
'configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.\n\n'
|
| 428 |
+
'An additional configuration "results" store all the aggregated results of the run.\n\n'
|
| 429 |
+
"To load the details from a run, you can for instance do the following:\n"
|
| 430 |
+
)
|
| 431 |
+
if self.general_config_tracker.model_source == "hf":
|
| 432 |
+
dataset_summary += "```python\nfrom datasets import load_dataset\n" f'data = load_dataset(\n\t"{repo_id}",\n\tname="{latest_model_name}",\n\tsplit="latest"\n)\n```\n\n'
|
| 433 |
+
dataset_summary += (
|
| 434 |
+
"## Latest results\n\n"
|
| 435 |
+
f'These are the [latest results from run {latest_datetime}]({last_results_file_path.replace("/resolve/", "/blob/")}) '
|
| 436 |
+
"(note that there might be results for other tasks in the repos if successive evals didn't cover the same tasks. "
|
| 437 |
+
'You find each in the results and the "latest" split for each eval):\n\n'
|
| 438 |
+
f"```python\n{results_string}\n```"
|
| 439 |
+
)
|
| 440 |
+
card_data = DatasetCardData(
|
| 441 |
+
dataset_summary=dataset_summary,
|
| 442 |
+
repo_url=f"https://huggingface.co/{self.general_config_tracker.model_name}",
|
| 443 |
+
pretty_name=f"Evaluation run of {self.general_config_tracker.model_name}",
|
| 444 |
+
leaderboard_url=self.leaderboard_url,
|
| 445 |
+
point_of_contact=self.point_of_contact,
|
| 446 |
+
)
|
| 447 |
+
card_metadata.to_dataset_card_data(card_data)
|
| 448 |
+
card = DatasetCard.from_template(
|
| 449 |
+
card_data,
|
| 450 |
+
pretty_name=card_data.pretty_name,
|
| 451 |
+
)
|
| 452 |
+
card.push_to_hub(repo_id, repo_type="dataset")
|
LaViDa/eval/lmms_eval/loggers/utils.py
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
import re
|
| 3 |
+
import subprocess
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
from typing import Any, Dict, Optional, Tuple, Union
|
| 6 |
+
|
| 7 |
+
import numpy as np
|
| 8 |
+
from loguru import logger
|
| 9 |
+
from torch.utils.collect_env import get_pretty_env_info
|
| 10 |
+
from transformers import __version__ as trans_version
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def remove_none_pattern(input_string: str) -> Tuple[str, bool]:
|
| 14 |
+
"""Remove the ',none' substring from the input_string if it exists at the end.
|
| 15 |
+
|
| 16 |
+
Args:
|
| 17 |
+
input_string (str): The input string from which to remove the ',none' substring.
|
| 18 |
+
|
| 19 |
+
Returns:
|
| 20 |
+
Tuple[str, bool]: A tuple containing the modified input_string with the ',none' substring removed
|
| 21 |
+
and a boolean indicating whether the modification was made (True) or not (False).
|
| 22 |
+
"""
|
| 23 |
+
# Define the pattern to match ',none' at the end of the string
|
| 24 |
+
pattern = re.compile(r",none$")
|
| 25 |
+
|
| 26 |
+
# Use sub() to replace ',none' with an empty string
|
| 27 |
+
result = re.sub(pattern, "", input_string)
|
| 28 |
+
|
| 29 |
+
# check if the input_string changed
|
| 30 |
+
removed = result != input_string
|
| 31 |
+
|
| 32 |
+
return result, removed
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def _handle_non_serializable(o: Any) -> Union[int, str, list]:
|
| 36 |
+
"""Handle non-serializable objects by converting them to serializable types.
|
| 37 |
+
|
| 38 |
+
Args:
|
| 39 |
+
o (Any): The object to be handled.
|
| 40 |
+
|
| 41 |
+
Returns:
|
| 42 |
+
Union[int, str, list]: The converted object. If the object is of type np.int64 or np.int32,
|
| 43 |
+
it will be converted to int. If the object is of type set, it will be converted
|
| 44 |
+
to a list. Otherwise, it will be converted to str.
|
| 45 |
+
"""
|
| 46 |
+
if isinstance(o, np.int64) or isinstance(o, np.int32):
|
| 47 |
+
return int(o)
|
| 48 |
+
elif isinstance(o, set):
|
| 49 |
+
return list(o)
|
| 50 |
+
else:
|
| 51 |
+
return str(o)
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def get_commit_from_path(repo_path: Union[Path, str]) -> Optional[str]:
|
| 55 |
+
try:
|
| 56 |
+
git_folder = Path(repo_path, ".git")
|
| 57 |
+
if git_folder.is_file():
|
| 58 |
+
git_folder = Path(
|
| 59 |
+
git_folder.parent,
|
| 60 |
+
git_folder.read_text(encoding="utf-8").split("\n")[0].split(" ")[-1],
|
| 61 |
+
)
|
| 62 |
+
if Path(git_folder, "HEAD").exists():
|
| 63 |
+
head_name = Path(git_folder, "HEAD").read_text(encoding="utf-8").split("\n")[0].split(" ")[-1]
|
| 64 |
+
head_ref = Path(git_folder, head_name)
|
| 65 |
+
git_hash = head_ref.read_text(encoding="utf-8").replace("\n", "")
|
| 66 |
+
else:
|
| 67 |
+
git_hash = None
|
| 68 |
+
except Exception as err:
|
| 69 |
+
logger.debug(f"Failed to retrieve a Git commit hash from path: {str(repo_path)}. Error: {err}")
|
| 70 |
+
return None
|
| 71 |
+
return git_hash
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def get_git_commit_hash():
|
| 75 |
+
"""
|
| 76 |
+
Gets the git commit hash of your current repo (if it exists).
|
| 77 |
+
Source: https://github.com/EleutherAI/gpt-neox/blob/b608043be541602170bfcfb8ec9bf85e8a0799e0/megatron/neox_arguments/neox_args.py#L42
|
| 78 |
+
"""
|
| 79 |
+
try:
|
| 80 |
+
git_hash = subprocess.check_output(["git", "describe", "--always"]).strip()
|
| 81 |
+
git_hash = git_hash.decode()
|
| 82 |
+
except (subprocess.CalledProcessError, FileNotFoundError):
|
| 83 |
+
# FileNotFoundError occurs when git not installed on system
|
| 84 |
+
git_hash = get_commit_from_path(os.getcwd()) # git hash of repo if exists
|
| 85 |
+
return git_hash
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def add_env_info(storage: Dict[str, Any]):
|
| 89 |
+
try:
|
| 90 |
+
pretty_env_info = get_pretty_env_info()
|
| 91 |
+
except Exception as err:
|
| 92 |
+
pretty_env_info = str(err)
|
| 93 |
+
transformers_version = trans_version
|
| 94 |
+
upper_dir_commit = get_commit_from_path(Path(os.getcwd(), "..")) # git hash of upper repo if exists
|
| 95 |
+
added_info = {
|
| 96 |
+
"pretty_env_info": pretty_env_info,
|
| 97 |
+
"transformers_version": transformers_version,
|
| 98 |
+
"upper_git_hash": upper_dir_commit, # in case this repo is submodule
|
| 99 |
+
}
|
| 100 |
+
storage.update(added_info)
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
def add_tokenizer_info(storage: Dict[str, Any], lm):
|
| 104 |
+
if getattr(lm, "tokenizer", False):
|
| 105 |
+
try:
|
| 106 |
+
tokenizer_info = {
|
| 107 |
+
"tokenizer_pad_token": [
|
| 108 |
+
lm.tokenizer.pad_token,
|
| 109 |
+
str(lm.tokenizer.pad_token_id),
|
| 110 |
+
],
|
| 111 |
+
"tokenizer_eos_token": [
|
| 112 |
+
lm.tokenizer.eos_token,
|
| 113 |
+
str(lm.tokenizer.eos_token_id),
|
| 114 |
+
],
|
| 115 |
+
"tokenizer_bos_token": [
|
| 116 |
+
lm.tokenizer.bos_token,
|
| 117 |
+
str(lm.tokenizer.bos_token_id),
|
| 118 |
+
],
|
| 119 |
+
"eot_token_id": getattr(lm, "eot_token_id", None),
|
| 120 |
+
"max_length": getattr(lm, "max_length", None),
|
| 121 |
+
}
|
| 122 |
+
storage.update(tokenizer_info)
|
| 123 |
+
except Exception as err:
|
| 124 |
+
logger.debug(f"Logging detailed tokenizer info failed with {err}, skipping...")
|
| 125 |
+
# seems gguf and textsynth do not have tokenizer
|
| 126 |
+
else:
|
| 127 |
+
logger.debug("LM does not have a 'tokenizer' attribute, not logging tokenizer metadata to results.")
|
LaViDa/eval/lmms_eval/loggers/wandb_logger.py
ADDED
|
@@ -0,0 +1,315 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import copy
|
| 2 |
+
import json
|
| 3 |
+
import logging
|
| 4 |
+
from typing import Any, Dict, List, Literal, Tuple
|
| 5 |
+
|
| 6 |
+
import numpy as np
|
| 7 |
+
import pandas as pd
|
| 8 |
+
from loguru import logger
|
| 9 |
+
from packaging.version import Version
|
| 10 |
+
|
| 11 |
+
from lmms_eval.loggers.utils import _handle_non_serializable, remove_none_pattern
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
def get_wandb_printer() -> Literal["Printer"]:
|
| 15 |
+
"""Returns a wandb printer instance for pretty stdout."""
|
| 16 |
+
from wandb.sdk.lib.printer import get_printer
|
| 17 |
+
from wandb.sdk.wandb_settings import Settings
|
| 18 |
+
|
| 19 |
+
printer = get_printer(Settings()._jupyter)
|
| 20 |
+
return printer
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
class WandbLogger:
|
| 24 |
+
def __init__(self, **kwargs) -> None:
|
| 25 |
+
"""Attaches to wandb logger if already initialized. Otherwise, passes kwargs to wandb.init()
|
| 26 |
+
|
| 27 |
+
Args:
|
| 28 |
+
kwargs Optional[Any]: Arguments for configuration.
|
| 29 |
+
|
| 30 |
+
Parse and log the results returned from evaluator.simple_evaluate() with:
|
| 31 |
+
wandb_logger.post_init(results)
|
| 32 |
+
wandb_logger.log_eval_result()
|
| 33 |
+
wandb_logger.log_eval_samples(results["samples"])
|
| 34 |
+
"""
|
| 35 |
+
try:
|
| 36 |
+
import wandb
|
| 37 |
+
|
| 38 |
+
assert Version(wandb.__version__) >= Version("0.13.6")
|
| 39 |
+
if Version(wandb.__version__) < Version("0.13.6"):
|
| 40 |
+
wandb.require("report-editing:v0")
|
| 41 |
+
except Exception as e:
|
| 42 |
+
logger.warning("To use the wandb reporting functionality please install wandb>=0.13.6.\n" "To install the latest version of wandb run `pip install wandb --upgrade`\n" f"{e}")
|
| 43 |
+
|
| 44 |
+
self.wandb_args: Dict[str, Any] = kwargs
|
| 45 |
+
|
| 46 |
+
# initialize a W&B run
|
| 47 |
+
if wandb.run is None:
|
| 48 |
+
self.run = wandb.init(**self.wandb_args)
|
| 49 |
+
else:
|
| 50 |
+
self.run = wandb.run
|
| 51 |
+
|
| 52 |
+
self.printer = get_wandb_printer()
|
| 53 |
+
|
| 54 |
+
def post_init(self, results: Dict[str, Any]) -> None:
|
| 55 |
+
self.results: Dict[str, Any] = copy.deepcopy(results)
|
| 56 |
+
self.task_names: List[str] = list(results.get("results", {}).keys())
|
| 57 |
+
self.group_names: List[str] = list(results.get("groups", {}).keys())
|
| 58 |
+
|
| 59 |
+
def _get_config(self) -> Dict[str, Any]:
|
| 60 |
+
"""Get configuration parameters."""
|
| 61 |
+
self.task_configs = self.results.get("configs", {})
|
| 62 |
+
cli_configs = self.results.get("config", {})
|
| 63 |
+
configs = {
|
| 64 |
+
"task_configs": self.task_configs,
|
| 65 |
+
"cli_configs": cli_configs,
|
| 66 |
+
}
|
| 67 |
+
|
| 68 |
+
return configs
|
| 69 |
+
|
| 70 |
+
def _sanitize_results_dict(self) -> Tuple[Dict[str, str], Dict[str, Any]]:
|
| 71 |
+
"""Sanitize the results dictionary."""
|
| 72 |
+
_results = copy.deepcopy(self.results.get("results", dict()))
|
| 73 |
+
|
| 74 |
+
# Remove None from the metric string name
|
| 75 |
+
tmp_results = copy.deepcopy(_results)
|
| 76 |
+
for task_name in self.task_names:
|
| 77 |
+
task_result = tmp_results.get(task_name, dict())
|
| 78 |
+
for metric_name, metric_value in task_result.items():
|
| 79 |
+
_metric_name, removed = remove_none_pattern(metric_name)
|
| 80 |
+
if removed:
|
| 81 |
+
_results[task_name][_metric_name] = metric_value
|
| 82 |
+
_results[task_name].pop(metric_name)
|
| 83 |
+
|
| 84 |
+
# remove string valued keys from the results dict
|
| 85 |
+
wandb_summary = {}
|
| 86 |
+
for task in self.task_names:
|
| 87 |
+
task_result = _results.get(task, dict())
|
| 88 |
+
for metric_name, metric_value in task_result.items():
|
| 89 |
+
if isinstance(metric_value, str):
|
| 90 |
+
wandb_summary[f"{task}/{metric_name}"] = metric_value
|
| 91 |
+
|
| 92 |
+
for summary_metric, summary_value in wandb_summary.items():
|
| 93 |
+
_task, _summary_metric = summary_metric.split("/")
|
| 94 |
+
_results[_task].pop(_summary_metric)
|
| 95 |
+
|
| 96 |
+
tmp_results = copy.deepcopy(_results)
|
| 97 |
+
for task_name, task_results in tmp_results.items():
|
| 98 |
+
for metric_name, metric_value in task_results.items():
|
| 99 |
+
_results[f"{task_name}/{metric_name}"] = metric_value
|
| 100 |
+
_results[task_name].pop(metric_name)
|
| 101 |
+
for task in self.task_names:
|
| 102 |
+
_results.pop(task)
|
| 103 |
+
|
| 104 |
+
return wandb_summary, _results
|
| 105 |
+
|
| 106 |
+
def _log_results_as_table(self) -> None:
|
| 107 |
+
"""Generate and log evaluation results as a table to W&B."""
|
| 108 |
+
columns = [
|
| 109 |
+
"Version",
|
| 110 |
+
"Filter",
|
| 111 |
+
"num_fewshot",
|
| 112 |
+
"Metric",
|
| 113 |
+
"Value",
|
| 114 |
+
"Stderr",
|
| 115 |
+
]
|
| 116 |
+
|
| 117 |
+
def make_table(columns: List[str], key: str = "results"):
|
| 118 |
+
import wandb
|
| 119 |
+
|
| 120 |
+
table = wandb.Table(columns=columns)
|
| 121 |
+
results = copy.deepcopy(self.results)
|
| 122 |
+
|
| 123 |
+
for k, dic in results.get(key).items():
|
| 124 |
+
if k in self.group_names and not key == "groups":
|
| 125 |
+
continue
|
| 126 |
+
version = results.get("versions").get(k)
|
| 127 |
+
if version == "N/A":
|
| 128 |
+
version = None
|
| 129 |
+
n = results.get("n-shot").get(k)
|
| 130 |
+
|
| 131 |
+
for (mf), v in dic.items():
|
| 132 |
+
m, _, f = mf.partition(",")
|
| 133 |
+
if m.endswith("_stderr"):
|
| 134 |
+
continue
|
| 135 |
+
if m == "alias":
|
| 136 |
+
continue
|
| 137 |
+
|
| 138 |
+
if m + "_stderr" + "," + f in dic:
|
| 139 |
+
se = dic[m + "_stderr" + "," + f]
|
| 140 |
+
if se != "N/A":
|
| 141 |
+
se = "%.4f" % se
|
| 142 |
+
table.add_data(*[k, version, f, n, m, str(v), str(se)])
|
| 143 |
+
else:
|
| 144 |
+
table.add_data(*[k, version, f, n, m, str(v), ""])
|
| 145 |
+
|
| 146 |
+
return table
|
| 147 |
+
|
| 148 |
+
# log the complete eval result to W&B Table
|
| 149 |
+
table = make_table(["Tasks"] + columns, "results")
|
| 150 |
+
self.run.log({"evaluation/eval_results": table})
|
| 151 |
+
|
| 152 |
+
if "groups" in self.results.keys():
|
| 153 |
+
table = make_table(["Groups"] + columns, "groups")
|
| 154 |
+
self.run.log({"evaluation/group_eval_results": table})
|
| 155 |
+
|
| 156 |
+
def _log_results_as_artifact(self) -> None:
|
| 157 |
+
"""Log results as JSON artifact to W&B."""
|
| 158 |
+
import wandb
|
| 159 |
+
|
| 160 |
+
dumped = json.dumps(self.results, indent=2, default=_handle_non_serializable, ensure_ascii=False)
|
| 161 |
+
artifact = wandb.Artifact("results", type="eval_results")
|
| 162 |
+
with artifact.new_file("results.json", mode="w", encoding="utf-8") as f:
|
| 163 |
+
f.write(dumped)
|
| 164 |
+
self.run.log_artifact(artifact)
|
| 165 |
+
|
| 166 |
+
def log_eval_result(self) -> None:
|
| 167 |
+
"""Log evaluation results to W&B."""
|
| 168 |
+
# Log configs to wandb
|
| 169 |
+
configs = self._get_config()
|
| 170 |
+
self.run.config.update(configs)
|
| 171 |
+
|
| 172 |
+
wandb_summary, self.wandb_results = self._sanitize_results_dict()
|
| 173 |
+
# update wandb.run.summary with items that were removed
|
| 174 |
+
self.run.summary.update(wandb_summary)
|
| 175 |
+
# Log the evaluation metrics to wandb
|
| 176 |
+
self.run.log(self.wandb_results)
|
| 177 |
+
# Log the evaluation metrics as W&B Table
|
| 178 |
+
self._log_results_as_table()
|
| 179 |
+
# Log the results dict as json to W&B Artifacts
|
| 180 |
+
self._log_results_as_artifact()
|
| 181 |
+
|
| 182 |
+
def _generate_dataset(self, data: List[Dict[str, Any]], config: Dict[str, Any]) -> pd.DataFrame:
|
| 183 |
+
"""Generate a dataset from evaluation data.
|
| 184 |
+
|
| 185 |
+
Args:
|
| 186 |
+
data (List[Dict[str, Any]]): The data to generate a dataset for.
|
| 187 |
+
config (Dict[str, Any]): The configuration of the task.
|
| 188 |
+
|
| 189 |
+
Returns:
|
| 190 |
+
pd.DataFrame: A dataframe that is ready to be uploaded to W&B.
|
| 191 |
+
"""
|
| 192 |
+
ids = [x["doc_id"] for x in data]
|
| 193 |
+
labels = [x["target"] for x in data]
|
| 194 |
+
instance = [""] * len(ids)
|
| 195 |
+
resps = [""] * len(ids)
|
| 196 |
+
filtered_resps = [""] * len(ids)
|
| 197 |
+
model_outputs = {}
|
| 198 |
+
|
| 199 |
+
metrics_list = config["metric_list"]
|
| 200 |
+
metrics = {}
|
| 201 |
+
for metric in metrics_list:
|
| 202 |
+
metric = metric.get("metric")
|
| 203 |
+
if metric in ["word_perplexity", "byte_perplexity", "bits_per_byte"]:
|
| 204 |
+
metrics[f"{metric}_loglikelihood"] = [x[metric][0] for x in data]
|
| 205 |
+
if metric in ["byte_perplexity", "bits_per_byte"]:
|
| 206 |
+
metrics[f"{metric}_bytes"] = [x[metric][1] for x in data]
|
| 207 |
+
else:
|
| 208 |
+
metrics[f"{metric}_words"] = [x[metric][1] for x in data]
|
| 209 |
+
else:
|
| 210 |
+
metrics[metric] = [x[metric] for x in data]
|
| 211 |
+
|
| 212 |
+
if config["output_type"] == "loglikelihood":
|
| 213 |
+
instance = [x["arguments"][0][0] for x in data]
|
| 214 |
+
labels = [x["arguments"][0][1] for x in data]
|
| 215 |
+
resps = [f'log probability of continuation is {x["resps"][0][0][0]} ' + "\n\n" + "continuation will {} generated with greedy sampling".format("not be" if not x["resps"][0][0][1] else "be") for x in data]
|
| 216 |
+
filtered_resps = [f'log probability of continuation is {x["filtered_resps"][0][0]} ' + "\n\n" + "continuation will {} generated with greedy sampling".format("not be" if not x["filtered_resps"][0][1] else "be") for x in data]
|
| 217 |
+
elif config["output_type"] == "multiple_choice":
|
| 218 |
+
instance = [x["arguments"][0][0] for x in data]
|
| 219 |
+
choices = ["\n".join([f"{idx}. {y[1]}" for idx, y in enumerate(x["arguments"])]) for x in data]
|
| 220 |
+
resps = [np.argmax([n[0][0] for n in x["resps"]]) for x in data]
|
| 221 |
+
filtered_resps = [np.argmax([n[0] for n in x["filtered_resps"]]) for x in data]
|
| 222 |
+
elif config["output_type"] == "loglikelihood_rolling":
|
| 223 |
+
instance = [x["arguments"][0][0] for x in data]
|
| 224 |
+
resps = [x["resps"][0][0] for x in data]
|
| 225 |
+
filtered_resps = [x["filtered_resps"][0] for x in data]
|
| 226 |
+
elif config["output_type"] == "generate_until":
|
| 227 |
+
instance = [x["arguments"][0][0] for x in data]
|
| 228 |
+
resps = [x["resps"][0][0] for x in data]
|
| 229 |
+
filtered_resps = [x["filtered_resps"][0] for x in data]
|
| 230 |
+
|
| 231 |
+
model_outputs["raw_predictions"] = resps
|
| 232 |
+
model_outputs["filtered_predictions"] = filtered_resps
|
| 233 |
+
|
| 234 |
+
df_data = {
|
| 235 |
+
"id": ids,
|
| 236 |
+
"data": instance,
|
| 237 |
+
}
|
| 238 |
+
if config["output_type"] == "multiple_choice":
|
| 239 |
+
df_data["choices"] = choices
|
| 240 |
+
|
| 241 |
+
tmp_data = {
|
| 242 |
+
"input_len": [len(x) for x in instance],
|
| 243 |
+
"labels": labels,
|
| 244 |
+
"output_type": config["output_type"],
|
| 245 |
+
}
|
| 246 |
+
df_data.update(tmp_data)
|
| 247 |
+
df_data.update(model_outputs)
|
| 248 |
+
df_data.update(metrics)
|
| 249 |
+
|
| 250 |
+
return pd.DataFrame(df_data)
|
| 251 |
+
|
| 252 |
+
def _log_samples_as_artifact(self, data: List[Dict[str, Any]], task_name: str) -> None:
|
| 253 |
+
import wandb
|
| 254 |
+
|
| 255 |
+
# log the samples as an artifact
|
| 256 |
+
dumped = json.dumps(
|
| 257 |
+
data,
|
| 258 |
+
indent=2,
|
| 259 |
+
default=_handle_non_serializable,
|
| 260 |
+
ensure_ascii=False,
|
| 261 |
+
)
|
| 262 |
+
artifact = wandb.Artifact(f"{task_name}", type="samples_by_task")
|
| 263 |
+
with artifact.new_file(f"{task_name}_eval_samples.json", mode="w", encoding="utf-8") as f:
|
| 264 |
+
f.write(dumped)
|
| 265 |
+
self.run.log_artifact(artifact)
|
| 266 |
+
# artifact.wait()
|
| 267 |
+
|
| 268 |
+
def log_eval_samples(self, samples: Dict[str, List[Dict[str, Any]]]) -> None:
|
| 269 |
+
"""Log evaluation samples to W&B.
|
| 270 |
+
|
| 271 |
+
Args:
|
| 272 |
+
samples (Dict[str, List[Dict[str, Any]]]): Evaluation samples for each task.
|
| 273 |
+
"""
|
| 274 |
+
task_names: List[str] = [x for x in self.task_names if x not in self.group_names]
|
| 275 |
+
|
| 276 |
+
ungrouped_tasks = []
|
| 277 |
+
tasks_by_groups = {}
|
| 278 |
+
|
| 279 |
+
for task_name in task_names:
|
| 280 |
+
group_names = self.task_configs[task_name].get("group", None)
|
| 281 |
+
if group_names:
|
| 282 |
+
if isinstance(group_names, str):
|
| 283 |
+
group_names = [group_names]
|
| 284 |
+
|
| 285 |
+
for group_name in group_names:
|
| 286 |
+
if not tasks_by_groups.get(group_name):
|
| 287 |
+
tasks_by_groups[group_name] = [task_name]
|
| 288 |
+
else:
|
| 289 |
+
tasks_by_groups[group_name].append(task_name)
|
| 290 |
+
else:
|
| 291 |
+
ungrouped_tasks.append(task_name)
|
| 292 |
+
|
| 293 |
+
for task_name in ungrouped_tasks:
|
| 294 |
+
eval_preds = samples[task_name]
|
| 295 |
+
|
| 296 |
+
# log the samples as a W&B Table
|
| 297 |
+
df = self._generate_dataset(eval_preds, self.task_configs.get(task_name))
|
| 298 |
+
self.run.log({f"{task_name}_eval_results": df})
|
| 299 |
+
|
| 300 |
+
# log the samples as a json file as W&B Artifact
|
| 301 |
+
self._log_samples_as_artifact(eval_preds, task_name)
|
| 302 |
+
|
| 303 |
+
for group, grouped_tasks in tasks_by_groups.items():
|
| 304 |
+
grouped_df = pd.DataFrame()
|
| 305 |
+
for task_name in grouped_tasks:
|
| 306 |
+
eval_preds = samples[task_name]
|
| 307 |
+
df = self._generate_dataset(eval_preds, self.task_configs.get(task_name))
|
| 308 |
+
df["group"] = group
|
| 309 |
+
df["task"] = task_name
|
| 310 |
+
grouped_df = pd.concat([grouped_df, df], ignore_index=True)
|
| 311 |
+
|
| 312 |
+
# log the samples as a json file as W&B Artifact
|
| 313 |
+
self._log_samples_as_artifact(eval_preds, task_name)
|
| 314 |
+
|
| 315 |
+
self.run.log({f"{group}_eval_results": grouped_df})
|
LaViDa/eval/lmms_eval/logging_utils.py
ADDED
|
@@ -0,0 +1,368 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Code mostly from: https://github.com/EleutherAI/lm-evaluation-harness/pull/1339, credit to: https://github.com/ayulockin
|
| 2 |
+
import copy
|
| 3 |
+
import glob
|
| 4 |
+
import json
|
| 5 |
+
import os
|
| 6 |
+
import re
|
| 7 |
+
from datetime import datetime
|
| 8 |
+
from typing import Any, Dict, List, Literal, Tuple, Union
|
| 9 |
+
|
| 10 |
+
import numpy as np
|
| 11 |
+
import pandas as pd
|
| 12 |
+
import tenacity
|
| 13 |
+
from loguru import logger
|
| 14 |
+
from packaging.version import Version
|
| 15 |
+
|
| 16 |
+
from lmms_eval import utils
|
| 17 |
+
|
| 18 |
+
try:
|
| 19 |
+
import wandb
|
| 20 |
+
|
| 21 |
+
assert Version(wandb.__version__) >= Version("0.13.6")
|
| 22 |
+
if Version(wandb.__version__) < Version("0.13.6"):
|
| 23 |
+
wandb.require("report-editing:v0")
|
| 24 |
+
except Exception as e:
|
| 25 |
+
logger.warning("To use the wandb reporting functionality please install wandb>=0.13.6.\n" "To install the latest version of wandb run `pip install wandb --upgrade`\n" f"{e}")
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def remove_none_pattern(input_string):
|
| 29 |
+
# Define the pattern to match ',none' at the end of the string
|
| 30 |
+
pattern = re.compile(r",none$")
|
| 31 |
+
|
| 32 |
+
# Use sub() to replace ',none' with an empty string
|
| 33 |
+
result = re.sub(pattern, "", input_string)
|
| 34 |
+
|
| 35 |
+
# check if the input_string changed
|
| 36 |
+
removed = result != input_string
|
| 37 |
+
|
| 38 |
+
return result, removed
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def _handle_non_serializable(o: Any) -> Union[int, str, list]:
|
| 42 |
+
"""Handle non-serializable objects by converting them to serializable types.
|
| 43 |
+
|
| 44 |
+
Args:
|
| 45 |
+
o (Any): The object to be handled.
|
| 46 |
+
|
| 47 |
+
Returns:
|
| 48 |
+
Union[int, str, list]: The converted object. If the object is of type np.int64 or np.int32,
|
| 49 |
+
it will be converted to int. If the object is of type set, it will be converted
|
| 50 |
+
to a list. Otherwise, it will be converted to str.
|
| 51 |
+
"""
|
| 52 |
+
if isinstance(o, np.int64) or isinstance(o, np.int32):
|
| 53 |
+
return int(o)
|
| 54 |
+
elif isinstance(o, set):
|
| 55 |
+
return list(o)
|
| 56 |
+
else:
|
| 57 |
+
return str(o)
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def get_wandb_printer() -> Literal["Printer"]:
|
| 61 |
+
"""Returns a wandb printer instance for pretty stdout."""
|
| 62 |
+
from wandb.sdk.lib.printer import get_printer
|
| 63 |
+
from wandb.sdk.wandb_settings import Settings
|
| 64 |
+
|
| 65 |
+
printer = get_printer(Settings()._jupyter)
|
| 66 |
+
return printer
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
# class WandbLogger:
|
| 70 |
+
class WandbLogger:
|
| 71 |
+
def __init__(self, args):
|
| 72 |
+
self.wandb_args = utils.simple_parse_args_string(args.wandb_args)
|
| 73 |
+
self.args = args
|
| 74 |
+
self.all_args_dict = vars(args)
|
| 75 |
+
self.printer = get_wandb_printer()
|
| 76 |
+
try:
|
| 77 |
+
self.init_run()
|
| 78 |
+
except Exception as e:
|
| 79 |
+
logger.warning(f"Failed to initialize W&B run: {e}")
|
| 80 |
+
os.environ["WANDB_MODE"] = "offline"
|
| 81 |
+
self.init_run()
|
| 82 |
+
|
| 83 |
+
def finish(self):
|
| 84 |
+
self.run.finish()
|
| 85 |
+
|
| 86 |
+
@tenacity.retry(wait=tenacity.wait_fixed(5), stop=tenacity.stop_after_attempt(5))
|
| 87 |
+
def init_run(self):
|
| 88 |
+
if "name" not in self.wandb_args:
|
| 89 |
+
if "config" in self.all_args_dict and self.all_args_dict["config"] != "":
|
| 90 |
+
self.wandb_args["name"] = self.all_args_dict["config"].split("/")[-1].replace(".yaml", "") + "/" + self.args.log_samples_suffix
|
| 91 |
+
else:
|
| 92 |
+
task_names = self.args.tasks.replace(",", "/")
|
| 93 |
+
self.wandb_args["name"] = f"{self.args.model}/<{task_names}>/{self.args.log_samples_suffix}"
|
| 94 |
+
if self.args.num_fewshot:
|
| 95 |
+
self.wandb_args["name"] += f"_{self.args.num_fewshot}shot"
|
| 96 |
+
if "project" not in self.wandb_args:
|
| 97 |
+
self.wandb_args["project"] = "lmms-eval"
|
| 98 |
+
# initialize a W&B run
|
| 99 |
+
self.run = wandb.init(**self.wandb_args)
|
| 100 |
+
|
| 101 |
+
def post_init(self, results: Dict[str, Any]) -> None:
|
| 102 |
+
self.results: Dict[str, Any] = copy.deepcopy(results)
|
| 103 |
+
self.task_names: List[str] = list(results.get("results", {}).keys())
|
| 104 |
+
self.group_names: List[str] = list(results.get("groups", {}).keys())
|
| 105 |
+
|
| 106 |
+
def _get_config(self) -> Dict[str, Any]:
|
| 107 |
+
"""Get configuration parameters."""
|
| 108 |
+
self.task_configs = self.results.get("configs", {})
|
| 109 |
+
cli_configs = self.results.get("config", {})
|
| 110 |
+
configs = {
|
| 111 |
+
"task_configs": self.task_configs,
|
| 112 |
+
"cli_configs": cli_configs,
|
| 113 |
+
}
|
| 114 |
+
|
| 115 |
+
return configs
|
| 116 |
+
|
| 117 |
+
def _sanitize_results_dict(self) -> Tuple[Dict[str, str], Dict[str, Any]]:
|
| 118 |
+
"""Sanitize the results dictionary."""
|
| 119 |
+
_results = copy.deepcopy(self.results.get("results", dict()))
|
| 120 |
+
_results["model_configs"] = self.results.get("model_configs", dict())
|
| 121 |
+
|
| 122 |
+
# Remove None from the metric string name
|
| 123 |
+
tmp_results = copy.deepcopy(_results)
|
| 124 |
+
for task_name in self.task_names:
|
| 125 |
+
task_result = tmp_results.get(task_name, dict())
|
| 126 |
+
for metric_name, metric_value in task_result.items():
|
| 127 |
+
_metric_name, removed = remove_none_pattern(metric_name)
|
| 128 |
+
if removed:
|
| 129 |
+
_results[task_name][_metric_name] = metric_value
|
| 130 |
+
_results[task_name].pop(metric_name)
|
| 131 |
+
|
| 132 |
+
# remove string valued keys from the results dict
|
| 133 |
+
wandb_summary = {}
|
| 134 |
+
for task in self.task_names:
|
| 135 |
+
task_result = _results.get(task, dict())
|
| 136 |
+
for metric_name, metric_value in task_result.items():
|
| 137 |
+
if isinstance(metric_value, str):
|
| 138 |
+
wandb_summary[f"{task}/{metric_name}"] = metric_value
|
| 139 |
+
|
| 140 |
+
wandb_summary["model_configs"] = self.results.get("model_configs", dict())
|
| 141 |
+
for summary_metric, summary_value in wandb_summary.items():
|
| 142 |
+
if summary_metric != "model_configs":
|
| 143 |
+
_task, _summary_metric = summary_metric.split("/")
|
| 144 |
+
_results[_task].pop(_summary_metric)
|
| 145 |
+
|
| 146 |
+
tmp_results = copy.deepcopy(_results)
|
| 147 |
+
for task_name, task_results in tmp_results.items():
|
| 148 |
+
if task_name != "model_configs":
|
| 149 |
+
for metric_name, metric_value in task_results.items():
|
| 150 |
+
_results[f"{task_name}/{metric_name}"] = metric_value
|
| 151 |
+
_results[task_name].pop(metric_name)
|
| 152 |
+
for task in self.task_names:
|
| 153 |
+
_results.pop(task)
|
| 154 |
+
|
| 155 |
+
return wandb_summary, _results
|
| 156 |
+
|
| 157 |
+
def _log_results_as_table(self) -> None:
|
| 158 |
+
"""Generate and log evaluation results as a table to W&B."""
|
| 159 |
+
columns = [
|
| 160 |
+
"Model",
|
| 161 |
+
"Args",
|
| 162 |
+
"Tasks",
|
| 163 |
+
"Version",
|
| 164 |
+
"Filter",
|
| 165 |
+
"num_fewshot",
|
| 166 |
+
"Metric",
|
| 167 |
+
"Value",
|
| 168 |
+
"Stderr",
|
| 169 |
+
]
|
| 170 |
+
|
| 171 |
+
def make_table(columns: List[str], key: str = "results"):
|
| 172 |
+
table = wandb.Table(columns=columns)
|
| 173 |
+
results = copy.deepcopy(self.results)
|
| 174 |
+
|
| 175 |
+
model_name = results.get("model_configs").get("model")
|
| 176 |
+
model_args = results.get("model_configs").get("model_args")
|
| 177 |
+
|
| 178 |
+
for k, dic in results.get(key).items():
|
| 179 |
+
if k in self.group_names and not key == "groups":
|
| 180 |
+
continue
|
| 181 |
+
version = results.get("versions").get(k)
|
| 182 |
+
if version == "N/A":
|
| 183 |
+
version = None
|
| 184 |
+
n = results.get("n-shot").get(k)
|
| 185 |
+
|
| 186 |
+
for (mf), v in dic.items():
|
| 187 |
+
m, _, f = mf.partition(",")
|
| 188 |
+
if m.endswith("_stderr"):
|
| 189 |
+
continue
|
| 190 |
+
if m == "alias":
|
| 191 |
+
continue
|
| 192 |
+
|
| 193 |
+
if m + "_stderr" + "," + f in dic:
|
| 194 |
+
se = dic[m + "_stderr" + "," + f]
|
| 195 |
+
if se != "N/A":
|
| 196 |
+
se = "%.4f" % se
|
| 197 |
+
data = [model_name, model_args, k, version, f, n, m, str(v), str(se)]
|
| 198 |
+
if key == "groups":
|
| 199 |
+
data = [self.group_names] + data
|
| 200 |
+
table.add_data(*data)
|
| 201 |
+
else:
|
| 202 |
+
data = [model_name, model_args, k, version, f, n, m, str(v), ""]
|
| 203 |
+
if key == "groups":
|
| 204 |
+
data = [self.group_names] + data
|
| 205 |
+
table.add_data(*data)
|
| 206 |
+
|
| 207 |
+
return table
|
| 208 |
+
|
| 209 |
+
# log the complete eval result to W&B Table
|
| 210 |
+
table = make_table(columns, "results")
|
| 211 |
+
self.run.log({"evaluation/eval_results": table})
|
| 212 |
+
|
| 213 |
+
if "groups" in self.results.keys():
|
| 214 |
+
table = make_table(["Groups"] + columns, "groups")
|
| 215 |
+
self.run.log({"evaluation/group_eval_results": table})
|
| 216 |
+
|
| 217 |
+
def _log_results_as_artifact(self) -> None:
|
| 218 |
+
"""Log results as JSON artifact to W&B."""
|
| 219 |
+
dumped = json.dumps(self.results, indent=2, default=_handle_non_serializable, ensure_ascii=False)
|
| 220 |
+
artifact = wandb.Artifact("results", type="eval_results")
|
| 221 |
+
with artifact.new_file("results.json", mode="w", encoding="utf-8") as f:
|
| 222 |
+
f.write(dumped)
|
| 223 |
+
self.run.log_artifact(artifact)
|
| 224 |
+
|
| 225 |
+
def log_eval_result(self) -> None:
|
| 226 |
+
"""Log evaluation results to W&B."""
|
| 227 |
+
# Log configs to wandb
|
| 228 |
+
configs = self._get_config()
|
| 229 |
+
self.run.config.update(configs, allow_val_change=True)
|
| 230 |
+
|
| 231 |
+
wandb_summary, self.wandb_results = self._sanitize_results_dict()
|
| 232 |
+
# update wandb.run.summary with items that were removed
|
| 233 |
+
self.run.summary.update(wandb_summary)
|
| 234 |
+
# Log the evaluation metrics to wandb
|
| 235 |
+
self.run.log(self.wandb_results)
|
| 236 |
+
# Log the evaluation metrics as W&B Table
|
| 237 |
+
self._log_results_as_table()
|
| 238 |
+
# Log the results dict as json to W&B Artifacts
|
| 239 |
+
self._log_results_as_artifact()
|
| 240 |
+
|
| 241 |
+
def _generate_dataset(self, data: List[Dict[str, Any]], config: Dict[str, Any]) -> pd.DataFrame:
|
| 242 |
+
"""Generate a dataset from evaluation data.
|
| 243 |
+
|
| 244 |
+
Args:
|
| 245 |
+
data (List[Dict[str, Any]]): The data to generate a dataset for.
|
| 246 |
+
config (Dict[str, Any]): The configuration of the task.
|
| 247 |
+
|
| 248 |
+
Returns:
|
| 249 |
+
pd.DataFrame: A dataframe that is ready to be uploaded to W&B.
|
| 250 |
+
"""
|
| 251 |
+
ids = [x["doc_id"] for x in data]
|
| 252 |
+
labels = [x["target"] for x in data]
|
| 253 |
+
instance = [""] * len(ids)
|
| 254 |
+
resps = [""] * len(ids)
|
| 255 |
+
filtered_resps = [""] * len(ids)
|
| 256 |
+
model_outputs = {}
|
| 257 |
+
|
| 258 |
+
metrics_list = config["metric_list"]
|
| 259 |
+
metrics = {}
|
| 260 |
+
for metric in metrics_list:
|
| 261 |
+
metric = metric.get("metric")
|
| 262 |
+
if metric in ["word_perplexity", "byte_perplexity", "bits_per_byte"]:
|
| 263 |
+
metrics[f"{metric}_loglikelihood"] = [x[metric][0] for x in data]
|
| 264 |
+
if metric in ["byte_perplexity", "bits_per_byte"]:
|
| 265 |
+
metrics[f"{metric}_bytes"] = [x[metric][1] for x in data]
|
| 266 |
+
else:
|
| 267 |
+
metrics[f"{metric}_words"] = [x[metric][1] for x in data]
|
| 268 |
+
else:
|
| 269 |
+
metrics[metric] = [x[metric] for x in data]
|
| 270 |
+
|
| 271 |
+
if config["output_type"] == "loglikelihood":
|
| 272 |
+
instance = [x["arguments"][0][0] for x in data]
|
| 273 |
+
labels = [x["arguments"][0][1] for x in data]
|
| 274 |
+
resps = [f'log probability of continuation is {x["resps"][0][0][0]} ' + "\n\n" + "continuation will {} generated with greedy sampling".format("not be" if not x["resps"][0][0][1] else "be") for x in data]
|
| 275 |
+
filtered_resps = [f'log probability of continuation is {x["filtered_resps"][0][0]} ' + "\n\n" + "continuation will {} generated with greedy sampling".format("not be" if not x["filtered_resps"][0][1] else "be") for x in data]
|
| 276 |
+
elif config["output_type"] == "multiple_choice":
|
| 277 |
+
instance = [x["arguments"][0][0] for x in data]
|
| 278 |
+
choices = ["\n".join([f"{idx}. {y[1]}" for idx, y in enumerate(x["arguments"])]) for x in data]
|
| 279 |
+
resps = [np.argmax([n[0][0] for n in x["resps"]]) for x in data]
|
| 280 |
+
filtered_resps = [np.argmax([n[0] for n in x["filtered_resps"]]) for x in data]
|
| 281 |
+
elif "generate_until" in config["output_type"]:
|
| 282 |
+
instance = [x["arguments"][0][0] for x in data]
|
| 283 |
+
resps = [x["resps"][0][0] for x in data]
|
| 284 |
+
filtered_resps = [x["filtered_resps"][0] for x in data]
|
| 285 |
+
|
| 286 |
+
model_outputs["raw_predictions"] = resps
|
| 287 |
+
model_outputs["filtered_predictions"] = filtered_resps
|
| 288 |
+
|
| 289 |
+
df_data = {
|
| 290 |
+
"id": ids,
|
| 291 |
+
"data": instance,
|
| 292 |
+
}
|
| 293 |
+
if config["output_type"] == "multiple_choice":
|
| 294 |
+
df_data["choices"] = choices
|
| 295 |
+
|
| 296 |
+
tmp_data = {
|
| 297 |
+
"input_len": [len(x) for x in instance],
|
| 298 |
+
"labels": labels,
|
| 299 |
+
"output_type": config["output_type"],
|
| 300 |
+
}
|
| 301 |
+
df_data.update(tmp_data)
|
| 302 |
+
df_data.update(model_outputs)
|
| 303 |
+
df_data.update(metrics)
|
| 304 |
+
|
| 305 |
+
return pd.DataFrame(df_data)
|
| 306 |
+
|
| 307 |
+
def _log_samples_as_artifact(self, data: List[Dict[str, Any]], task_name: str) -> None:
|
| 308 |
+
# log the samples as an artifact
|
| 309 |
+
dumped = json.dumps(
|
| 310 |
+
data,
|
| 311 |
+
indent=2,
|
| 312 |
+
default=_handle_non_serializable,
|
| 313 |
+
ensure_ascii=False,
|
| 314 |
+
)
|
| 315 |
+
artifact = wandb.Artifact(f"{task_name}", type="samples_by_task")
|
| 316 |
+
with artifact.new_file(f"{task_name}_eval_samples.json", mode="w", encoding="utf-8") as f:
|
| 317 |
+
f.write(dumped)
|
| 318 |
+
self.run.log_artifact(artifact)
|
| 319 |
+
# artifact.wait()
|
| 320 |
+
|
| 321 |
+
def log_eval_samples(self, samples: Dict[str, List[Dict[str, Any]]]) -> None:
|
| 322 |
+
"""Log evaluation samples to W&B.
|
| 323 |
+
|
| 324 |
+
Args:
|
| 325 |
+
samples (Dict[str, List[Dict[str, Any]]]): Evaluation samples for each task.
|
| 326 |
+
"""
|
| 327 |
+
task_names: List[str] = [x for x in self.task_names if x not in self.group_names]
|
| 328 |
+
|
| 329 |
+
ungrouped_tasks = []
|
| 330 |
+
tasks_by_groups = {}
|
| 331 |
+
|
| 332 |
+
for task_name in task_names:
|
| 333 |
+
group_names = self.task_configs[task_name].get("group", None)
|
| 334 |
+
if group_names:
|
| 335 |
+
if isinstance(group_names, str):
|
| 336 |
+
group_names = [group_names]
|
| 337 |
+
|
| 338 |
+
for group_name in group_names:
|
| 339 |
+
if not tasks_by_groups.get(group_name):
|
| 340 |
+
tasks_by_groups[group_name] = [task_name]
|
| 341 |
+
else:
|
| 342 |
+
tasks_by_groups[group_name].append(task_name)
|
| 343 |
+
else:
|
| 344 |
+
ungrouped_tasks.append(task_name)
|
| 345 |
+
|
| 346 |
+
for task_name in ungrouped_tasks:
|
| 347 |
+
eval_preds = samples[task_name]
|
| 348 |
+
|
| 349 |
+
# log the samples as a W&B Table
|
| 350 |
+
df = self._generate_dataset(eval_preds, self.task_configs.get(task_name))
|
| 351 |
+
self.run.log({f"{task_name}_eval_results": df})
|
| 352 |
+
|
| 353 |
+
# log the samples as a json file as W&B Artifact
|
| 354 |
+
self._log_samples_as_artifact(eval_preds, task_name)
|
| 355 |
+
|
| 356 |
+
for group, grouped_tasks in tasks_by_groups.items():
|
| 357 |
+
grouped_df = pd.DataFrame()
|
| 358 |
+
for task_name in grouped_tasks:
|
| 359 |
+
eval_preds = samples[task_name]
|
| 360 |
+
df = self._generate_dataset(eval_preds, self.task_configs.get(task_name))
|
| 361 |
+
df["group"] = group
|
| 362 |
+
df["task"] = task_name
|
| 363 |
+
grouped_df = pd.concat([grouped_df, df], ignore_index=True)
|
| 364 |
+
|
| 365 |
+
# log the samples as a json file as W&B Artifact
|
| 366 |
+
self._log_samples_as_artifact(eval_preds, task_name)
|
| 367 |
+
|
| 368 |
+
self.run.log({f"{group}_eval_results": grouped_df})
|
LaViDa/eval/lmms_eval/models/egogpt.py
ADDED
|
@@ -0,0 +1,472 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import copy
|
| 2 |
+
import json
|
| 3 |
+
import logging
|
| 4 |
+
import math
|
| 5 |
+
import re
|
| 6 |
+
import warnings
|
| 7 |
+
from datetime import timedelta
|
| 8 |
+
from typing import List, Optional, Tuple, Union
|
| 9 |
+
|
| 10 |
+
import numpy as np
|
| 11 |
+
import PIL
|
| 12 |
+
import torch
|
| 13 |
+
import transformers
|
| 14 |
+
from accelerate import Accelerator, DistributedType, InitProcessGroupKwargs
|
| 15 |
+
from accelerate.state import AcceleratorState
|
| 16 |
+
from decord import VideoReader, cpu
|
| 17 |
+
from packaging import version
|
| 18 |
+
from tqdm import tqdm
|
| 19 |
+
from transformers import AutoConfig
|
| 20 |
+
|
| 21 |
+
from lmms_eval import utils
|
| 22 |
+
from lmms_eval.api.instance import Instance
|
| 23 |
+
from lmms_eval.api.model import lmms
|
| 24 |
+
from lmms_eval.api.registry import register_model
|
| 25 |
+
|
| 26 |
+
# Suppress warnings
|
| 27 |
+
warnings.filterwarnings("ignore")
|
| 28 |
+
|
| 29 |
+
# Configure logging
|
| 30 |
+
eval_logger = logging.getLogger("lmms-eval")
|
| 31 |
+
|
| 32 |
+
# Enable TF32 for CUDA
|
| 33 |
+
torch.backends.cuda.matmul.allow_tf32 = True
|
| 34 |
+
|
| 35 |
+
# Import LLaVA modules
|
| 36 |
+
try:
|
| 37 |
+
import copy
|
| 38 |
+
import os
|
| 39 |
+
import re
|
| 40 |
+
import sys
|
| 41 |
+
import warnings
|
| 42 |
+
|
| 43 |
+
import numpy as np
|
| 44 |
+
import requests
|
| 45 |
+
import soundfile as sf
|
| 46 |
+
import torch
|
| 47 |
+
import whisper
|
| 48 |
+
from decord import VideoReader, cpu
|
| 49 |
+
from egogpt.constants import (
|
| 50 |
+
DEFAULT_IMAGE_TOKEN,
|
| 51 |
+
DEFAULT_SPEECH_TOKEN,
|
| 52 |
+
IGNORE_INDEX,
|
| 53 |
+
IMAGE_TOKEN_INDEX,
|
| 54 |
+
SPEECH_TOKEN_INDEX,
|
| 55 |
+
)
|
| 56 |
+
from egogpt.conversation import SeparatorStyle, conv_templates
|
| 57 |
+
from egogpt.mm_utils import get_model_name_from_path, process_images
|
| 58 |
+
from egogpt.model.builder import load_pretrained_model
|
| 59 |
+
from PIL import Image
|
| 60 |
+
from scipy.signal import resample
|
| 61 |
+
except ImportError as e:
|
| 62 |
+
eval_logger.debug(f"egogpt is not installed. Please install egogpt to use this model.\nError: {e}")
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
# Determine best attention implementation
|
| 66 |
+
if version.parse(torch.__version__) >= version.parse("2.1.2"):
|
| 67 |
+
best_fit_attn_implementation = "sdpa"
|
| 68 |
+
else:
|
| 69 |
+
best_fit_attn_implementation = "eager"
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
@register_model("egogpt")
|
| 73 |
+
class EgoGPT(lmms):
|
| 74 |
+
"""
|
| 75 |
+
EgoGPT Model
|
| 76 |
+
"""
|
| 77 |
+
|
| 78 |
+
def __init__(
|
| 79 |
+
self,
|
| 80 |
+
pretrained: str = "checkpoints/egogpt_IT_12k_1126_zero3",
|
| 81 |
+
truncation: Optional[bool] = True,
|
| 82 |
+
device: Optional[str] = "cuda:0",
|
| 83 |
+
batch_size: Optional[Union[int, str]] = 1,
|
| 84 |
+
model_name: Optional[str] = None,
|
| 85 |
+
attn_implementation: Optional[str] = best_fit_attn_implementation,
|
| 86 |
+
device_map: Optional[str] = "cuda:0",
|
| 87 |
+
conv_template: Optional[str] = "qwen_1_5",
|
| 88 |
+
use_cache: Optional[bool] = True,
|
| 89 |
+
truncate_context: Optional[bool] = False, # whether to truncate the context in generation, set it False for LLaVA-1.6
|
| 90 |
+
customized_config: Optional[str] = None, # ends in json
|
| 91 |
+
max_frames_num: Optional[int] = 32,
|
| 92 |
+
mm_spatial_pool_stride: Optional[int] = 2,
|
| 93 |
+
mm_spatial_pool_mode: Optional[str] = "bilinear",
|
| 94 |
+
token_strategy: Optional[str] = "single", # could be "single" or "multiple", "multiple" denotes adding multiple <image> tokens for each frame
|
| 95 |
+
video_decode_backend: str = "decord",
|
| 96 |
+
**kwargs,
|
| 97 |
+
) -> None:
|
| 98 |
+
super().__init__()
|
| 99 |
+
# Do not use kwargs for now
|
| 100 |
+
assert kwargs == {}, f"Unexpected kwargs: {kwargs}"
|
| 101 |
+
|
| 102 |
+
accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52))
|
| 103 |
+
accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs])
|
| 104 |
+
if accelerator.num_processes > 1:
|
| 105 |
+
self._device = torch.device(f"cuda:{accelerator.local_process_index}")
|
| 106 |
+
self.device_map = f"cuda:{accelerator.local_process_index}"
|
| 107 |
+
elif accelerator.num_processes == 1 and device_map == "auto":
|
| 108 |
+
self._device = torch.device(device)
|
| 109 |
+
self.device_map = device_map
|
| 110 |
+
else:
|
| 111 |
+
self._device = torch.device(f"cuda:{accelerator.local_process_index}")
|
| 112 |
+
self.device_map = f"cuda:{accelerator.local_process_index}"
|
| 113 |
+
|
| 114 |
+
egogpt_model_args = {}
|
| 115 |
+
if attn_implementation is not None:
|
| 116 |
+
egogpt_model_args["attn_implementation"] = attn_implementation
|
| 117 |
+
|
| 118 |
+
self.pretrained = pretrained
|
| 119 |
+
self.token_strategy = token_strategy
|
| 120 |
+
self.max_frames_num = max_frames_num
|
| 121 |
+
self.mm_spatial_pool_stride = mm_spatial_pool_stride
|
| 122 |
+
self.mm_spatial_pool_mode = mm_spatial_pool_mode
|
| 123 |
+
self.video_decode_backend = video_decode_backend
|
| 124 |
+
# Try to load the model with the multimodal argument
|
| 125 |
+
self._tokenizer, self._model, self._max_length = load_pretrained_model(pretrained, device_map=self.device_map, **egogpt_model_args)
|
| 126 |
+
self._image_processor = self._model.get_vision_tower().image_processor
|
| 127 |
+
self._config = self._model.config
|
| 128 |
+
self.model.eval()
|
| 129 |
+
self.truncation = truncation
|
| 130 |
+
self.batch_size_per_gpu = int(batch_size)
|
| 131 |
+
self.conv_template = conv_template
|
| 132 |
+
self.use_cache = use_cache
|
| 133 |
+
self.truncate_context = truncate_context
|
| 134 |
+
assert self.batch_size_per_gpu == 1
|
| 135 |
+
|
| 136 |
+
if accelerator.num_processes > 1:
|
| 137 |
+
assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported."
|
| 138 |
+
# If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model
|
| 139 |
+
# Also, you have to select zero stage 0 (equivalent to DDP) in order to make the prepare model works
|
| 140 |
+
# I tried to set different parameters in the kwargs to let default zero 2 stage works, but it didn't work.
|
| 141 |
+
if accelerator.distributed_type == DistributedType.DEEPSPEED:
|
| 142 |
+
kwargs = {
|
| 143 |
+
"train_micro_batch_size_per_gpu": self.batch_size_per_gpu,
|
| 144 |
+
"train_batch_size": self.batch_size_per_gpu * accelerator.num_processes,
|
| 145 |
+
}
|
| 146 |
+
AcceleratorState().deepspeed_plugin.deepspeed_config_process(must_match=True, **kwargs)
|
| 147 |
+
eval_logger.info("Detected that you are using DistributedType.DEEPSPEED. Make sure you run `accelerate config` and set zero stage to 0")
|
| 148 |
+
|
| 149 |
+
if accelerator.distributed_type == DistributedType.FSDP or accelerator.distributed_type == DistributedType.DEEPSPEED:
|
| 150 |
+
self._model = accelerator.prepare(self.model)
|
| 151 |
+
else:
|
| 152 |
+
self._model = accelerator.prepare_model(self.model, evaluation_mode=True)
|
| 153 |
+
self.accelerator = accelerator
|
| 154 |
+
if self.accelerator.is_local_main_process:
|
| 155 |
+
eval_logger.info(f"Using {accelerator.num_processes} devices with data parallelism")
|
| 156 |
+
self._rank = self.accelerator.local_process_index
|
| 157 |
+
self._world_size = self.accelerator.num_processes
|
| 158 |
+
|
| 159 |
+
elif accelerator.num_processes == 1 and device_map == "auto":
|
| 160 |
+
eval_logger.info(f"Using {accelerator.num_processes} devices with tensor parallelism")
|
| 161 |
+
self._rank = 0
|
| 162 |
+
self._world_size = 1
|
| 163 |
+
|
| 164 |
+
else:
|
| 165 |
+
eval_logger.info(f"Using single device: {self._device}")
|
| 166 |
+
self.model.to(self._device)
|
| 167 |
+
self._rank = 0
|
| 168 |
+
self._world_size = 1
|
| 169 |
+
|
| 170 |
+
@property
|
| 171 |
+
def config(self):
|
| 172 |
+
# return the associated transformers.AutoConfig for the given pretrained model.
|
| 173 |
+
return self._config
|
| 174 |
+
|
| 175 |
+
@property
|
| 176 |
+
def tokenizer(self):
|
| 177 |
+
return self._tokenizer
|
| 178 |
+
|
| 179 |
+
@property
|
| 180 |
+
def model(self):
|
| 181 |
+
# returns the model, unwrapping it if using Accelerate
|
| 182 |
+
if hasattr(self, "accelerator"):
|
| 183 |
+
return self.accelerator.unwrap_model(self._model)
|
| 184 |
+
else:
|
| 185 |
+
return self._model
|
| 186 |
+
|
| 187 |
+
@property
|
| 188 |
+
def eot_token_id(self):
|
| 189 |
+
# we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
|
| 190 |
+
return self.tokenizer.eos_token_id
|
| 191 |
+
|
| 192 |
+
@property
|
| 193 |
+
def max_length(self):
|
| 194 |
+
return self._max_length
|
| 195 |
+
|
| 196 |
+
def pad_sequence(self, input_ids, batch_first, padding_value):
|
| 197 |
+
if self.tokenizer.padding_side == "left":
|
| 198 |
+
input_ids = [torch.flip(_input_ids, [0]) for _input_ids in input_ids]
|
| 199 |
+
input_ids = torch.nn.utils.rnn.pad_sequence(input_ids, batch_first=batch_first, padding_value=padding_value)
|
| 200 |
+
if self.tokenizer.padding_side == "left":
|
| 201 |
+
input_ids = torch.flip(input_ids, [1])
|
| 202 |
+
return input_ids
|
| 203 |
+
|
| 204 |
+
@property
|
| 205 |
+
def batch_size(self):
|
| 206 |
+
return self.batch_size_per_gpu
|
| 207 |
+
|
| 208 |
+
@property
|
| 209 |
+
def device(self):
|
| 210 |
+
return self._device
|
| 211 |
+
|
| 212 |
+
@property
|
| 213 |
+
def rank(self):
|
| 214 |
+
return self._rank
|
| 215 |
+
|
| 216 |
+
@property
|
| 217 |
+
def world_size(self):
|
| 218 |
+
return self._world_size
|
| 219 |
+
|
| 220 |
+
def tok_encode(self, string: str, left_truncate_len=None, add_special_tokens=None) -> List[int]:
|
| 221 |
+
""" """
|
| 222 |
+
add_special_tokens = False if add_special_tokens is None else add_special_tokens
|
| 223 |
+
encoding = self.tokenizer.encode(string, add_special_tokens=add_special_tokens)
|
| 224 |
+
# left-truncate the encoded context to be at most `left_truncate_len` tokens long
|
| 225 |
+
if left_truncate_len:
|
| 226 |
+
encoding = encoding[-left_truncate_len:]
|
| 227 |
+
return encoding
|
| 228 |
+
|
| 229 |
+
def tok_decode(self, tokens):
|
| 230 |
+
try:
|
| 231 |
+
return self.tokenizer.decode(tokens)
|
| 232 |
+
except:
|
| 233 |
+
return self.tokenizer.decode([tokens])
|
| 234 |
+
|
| 235 |
+
def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
|
| 236 |
+
raise NotImplementedError("Loglikelihood is not implemented for EgoGPT")
|
| 237 |
+
|
| 238 |
+
def flatten(self, input):
|
| 239 |
+
new_list = []
|
| 240 |
+
for i in input:
|
| 241 |
+
for j in i:
|
| 242 |
+
new_list.append(j)
|
| 243 |
+
return new_list
|
| 244 |
+
|
| 245 |
+
def split_text(self, text, keywords):
|
| 246 |
+
pattern = "(" + "|".join(map(re.escape, keywords)) + ")"
|
| 247 |
+
parts = re.split(pattern, text)
|
| 248 |
+
parts = [part for part in parts if part]
|
| 249 |
+
return parts
|
| 250 |
+
|
| 251 |
+
def load_video(self, video_path=None, audio_path=None, max_frames_num=16, fps=1, task_name=None):
|
| 252 |
+
if audio_path is not None:
|
| 253 |
+
speech, sample_rate = sf.read(audio_path)
|
| 254 |
+
if sample_rate != 16000:
|
| 255 |
+
target_length = int(len(speech) * 16000 / sample_rate)
|
| 256 |
+
speech = resample(speech, target_length)
|
| 257 |
+
if speech.ndim > 1:
|
| 258 |
+
speech = np.mean(speech, axis=1)
|
| 259 |
+
# max_length = 480000
|
| 260 |
+
speech = whisper.pad_or_trim(speech.astype(np.float32))
|
| 261 |
+
speech = whisper.log_mel_spectrogram(speech, n_mels=128).permute(1, 0)
|
| 262 |
+
speech_lengths = torch.LongTensor([speech.shape[0]])
|
| 263 |
+
else:
|
| 264 |
+
speech = torch.zeros(3000, 128)
|
| 265 |
+
speech_lengths = torch.LongTensor([3000])
|
| 266 |
+
|
| 267 |
+
vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)
|
| 268 |
+
total_frame_num = len(vr)
|
| 269 |
+
avg_fps = round(vr.get_avg_fps() / fps)
|
| 270 |
+
frame_idx = [i for i in range(0, total_frame_num, avg_fps)]
|
| 271 |
+
frame_time = [i / avg_fps for i in frame_idx]
|
| 272 |
+
|
| 273 |
+
if max_frames_num > 0:
|
| 274 |
+
if len(frame_idx) > max_frames_num:
|
| 275 |
+
uniform_sampled_frames = np.linspace(0, total_frame_num - 1, max_frames_num, dtype=int)
|
| 276 |
+
frame_idx = uniform_sampled_frames.tolist()
|
| 277 |
+
if task_name == "egoplan":
|
| 278 |
+
# add current ovservation frame
|
| 279 |
+
frame_idx.append(total_frame_num - 1)
|
| 280 |
+
video = vr.get_batch(frame_idx).asnumpy()
|
| 281 |
+
return video, speech, speech_lengths
|
| 282 |
+
|
| 283 |
+
def generate_until(self, requests: List[Instance]) -> List[str]:
|
| 284 |
+
res = []
|
| 285 |
+
|
| 286 |
+
def _collate(x):
|
| 287 |
+
# the negative sign on len(toks) sorts descending - this has a few advantages:
|
| 288 |
+
# - time estimates will always be over not underestimates, which is more useful for planning
|
| 289 |
+
# - to know the size of a batch when going through the list, you know the first one is always the batch
|
| 290 |
+
# padded context length. this is useful to simplify the batching logic and more importantly to make
|
| 291 |
+
# automatic adaptive batches much much easier to implement
|
| 292 |
+
# - any OOMs will happen right away rather than near the end
|
| 293 |
+
toks = self.tok_encode(x[0])
|
| 294 |
+
return -len(toks), x[0]
|
| 295 |
+
|
| 296 |
+
# we group requests by their generation_kwargs,
|
| 297 |
+
# so that we don't try to execute e.g. greedy sampling and temp=0.8 sampling
|
| 298 |
+
# in the same batch.
|
| 299 |
+
metadata = requests[0].metadata
|
| 300 |
+
re_ords = utils.Collator([reg.args for reg in requests], _collate, grouping=True)
|
| 301 |
+
chunks = re_ords.get_batched(n=self.batch_size, batch_fn=None)
|
| 302 |
+
num_iters = len(requests) // self.batch_size if len(requests) % self.batch_size == 0 else len(requests) // self.batch_size + 1
|
| 303 |
+
pbar = tqdm(total=num_iters, disable=(self.rank != 0), desc="Model Responding")
|
| 304 |
+
|
| 305 |
+
origin_image_aspect_ratio = getattr(self._config, "image_aspect_ratio", None)
|
| 306 |
+
|
| 307 |
+
for chunk in chunks:
|
| 308 |
+
batched_contexts, all_gen_kwargs, batched_doc_to_visual, batched_doc_id, batched_task, batched_split = zip(*chunk)
|
| 309 |
+
task = batched_task[0]
|
| 310 |
+
split = batched_split[0]
|
| 311 |
+
batched_visuals = [batched_doc_to_visual[0](self.task_dict[task][split][ids]) for ids in batched_doc_id] # [B, N]
|
| 312 |
+
assert len(batched_visuals) == 1
|
| 313 |
+
|
| 314 |
+
# we assume all gen kwargs in the batch are the same
|
| 315 |
+
# this is safe to assume because the `grouper` object ensures it.
|
| 316 |
+
gen_kwargs = all_gen_kwargs[0]
|
| 317 |
+
if "until" in gen_kwargs:
|
| 318 |
+
gen_kwargs.pop("until")
|
| 319 |
+
|
| 320 |
+
question_input = []
|
| 321 |
+
# import ipdb; ipdb.set_trace()
|
| 322 |
+
for visual, context in zip(batched_visuals, batched_contexts):
|
| 323 |
+
if origin_image_aspect_ratio is not None and self._config.image_aspect_ratio != origin_image_aspect_ratio:
|
| 324 |
+
self._config.image_aspect_ratio = origin_image_aspect_ratio
|
| 325 |
+
eval_logger.info(f"Resetting image aspect ratio to {origin_image_aspect_ratio}")
|
| 326 |
+
|
| 327 |
+
if visual is None or visual == []: # for text-only tasks.
|
| 328 |
+
visual = None
|
| 329 |
+
task_type = "text"
|
| 330 |
+
placeholder_count = 0
|
| 331 |
+
image_tensor = None
|
| 332 |
+
else:
|
| 333 |
+
if len(visual) > 1 or "image_aspect_ratio" not in self._config.__dict__: # for multi image case, we treat per image aspect ratio as "pad" by default.
|
| 334 |
+
self._config.image_aspect_ratio = getattr(gen_kwargs, "image_aspect_ratio", "pad")
|
| 335 |
+
eval_logger.info(f"In Multi-Image setting, image aspect ratio: {self._config.image_aspect_ratio}")
|
| 336 |
+
|
| 337 |
+
if "task_type" in metadata and metadata["task_type"] == "video" and "sample_frames" in metadata: # overwrite logic for video task with multiple static image frames
|
| 338 |
+
assert type(visual) == list, "sample_frames must be specified for video task"
|
| 339 |
+
sample_indices = np.linspace(0, len(visual) - 1, metadata["sample_frames"], dtype=int)
|
| 340 |
+
visual = [visual[i] for i in sample_indices]
|
| 341 |
+
assert len(visual) == metadata["sample_frames"]
|
| 342 |
+
|
| 343 |
+
image_tensor = process_images(visual, self._image_processor, self._config)
|
| 344 |
+
if type(image_tensor) is list:
|
| 345 |
+
image_tensor = [_image.to(dtype=torch.float16, device=self.device) for _image in image_tensor]
|
| 346 |
+
else:
|
| 347 |
+
image_tensor = image_tensor.to(dtype=torch.float16, device=self.device)
|
| 348 |
+
image_tensor = [image_tensor]
|
| 349 |
+
task_type = "video"
|
| 350 |
+
placeholder_count = 1
|
| 351 |
+
|
| 352 |
+
elif type(visual[0]) == PIL.Image.Image: # For image, multi-image tasks
|
| 353 |
+
image_tensor = process_images(visual, self._image_processor, self._config)
|
| 354 |
+
speech = torch.zeros(3000, 128)
|
| 355 |
+
speech_lengths = torch.LongTensor([3000])
|
| 356 |
+
if type(image_tensor) is list:
|
| 357 |
+
image_tensor = [_image.to(dtype=torch.float16, device=self.device) for _image in image_tensor]
|
| 358 |
+
else:
|
| 359 |
+
image_tensor = image_tensor.to(dtype=torch.float16, device=self.device)
|
| 360 |
+
|
| 361 |
+
task_type = "image"
|
| 362 |
+
placeholder_count = len(visual) if isinstance(visual, list) else 1
|
| 363 |
+
|
| 364 |
+
elif type(visual[0]) == str: # For video task
|
| 365 |
+
image_tensor = []
|
| 366 |
+
try:
|
| 367 |
+
if self.video_decode_backend == "decord":
|
| 368 |
+
if "egoplan" in visual[0]:
|
| 369 |
+
task_name = "egoplan"
|
| 370 |
+
else:
|
| 371 |
+
task_name = None
|
| 372 |
+
frames, speech, speech_lengths = self.load_video(video_path=visual[0], max_frames_num=self.max_frames_num, task_name=task_name)
|
| 373 |
+
else:
|
| 374 |
+
raise NotImplementedError("Only decord backend is supported for video task")
|
| 375 |
+
processed_frames = self._image_processor.preprocess(frames, return_tensors="pt")["pixel_values"].half().cuda()
|
| 376 |
+
processed_frames = processed_frames.half()
|
| 377 |
+
image_tensor.append(processed_frames)
|
| 378 |
+
image_sizes = [frames[0].size]
|
| 379 |
+
except Exception as e:
|
| 380 |
+
eval_logger.error(f"Error {e} in loading video")
|
| 381 |
+
image_tensor = None
|
| 382 |
+
|
| 383 |
+
task_type = "video"
|
| 384 |
+
placeholder_count = len(frames) if self.token_strategy == "multiple" else 1
|
| 385 |
+
if DEFAULT_IMAGE_TOKEN not in context:
|
| 386 |
+
question = DEFAULT_IMAGE_TOKEN + "\n" + context
|
| 387 |
+
else:
|
| 388 |
+
question = context
|
| 389 |
+
speech = torch.stack([speech]).to(self.device).half()
|
| 390 |
+
# This is much safer for llama3, as we now have some object type in it
|
| 391 |
+
if "llama_3" in self.conv_template:
|
| 392 |
+
conv = copy.deepcopy(conv_templates[self.conv_template])
|
| 393 |
+
else:
|
| 394 |
+
conv = conv_templates[self.conv_template].copy()
|
| 395 |
+
|
| 396 |
+
if utils.is_json(question): # conversational question input
|
| 397 |
+
question = json.loads(question)
|
| 398 |
+
for idx, item in enumerate(question):
|
| 399 |
+
role = conv.roles[idx % 2]
|
| 400 |
+
message = item["value"]
|
| 401 |
+
conv.append_message(role, message)
|
| 402 |
+
|
| 403 |
+
assert len(conv.messages) % 2 == 1
|
| 404 |
+
conv.append_message(conv.roles[1], None)
|
| 405 |
+
prompt_question = conv.get_prompt()
|
| 406 |
+
question_input.append(prompt_question)
|
| 407 |
+
else: # only simple string for question
|
| 408 |
+
conv.append_message(conv.roles[0], question)
|
| 409 |
+
conv.append_message(conv.roles[1], None)
|
| 410 |
+
prompt_question = conv.get_prompt()
|
| 411 |
+
question_input.append(prompt_question)
|
| 412 |
+
|
| 413 |
+
# preconfigure gen_kwargs with defaults
|
| 414 |
+
if "max_new_tokens" not in gen_kwargs:
|
| 415 |
+
gen_kwargs["max_new_tokens"] = 1024
|
| 416 |
+
if "temperature" not in gen_kwargs:
|
| 417 |
+
gen_kwargs["temperature"] = 0
|
| 418 |
+
if "do_sample" not in gen_kwargs:
|
| 419 |
+
gen_kwargs["do_sample"] = False
|
| 420 |
+
if "top_p" not in gen_kwargs:
|
| 421 |
+
gen_kwargs["top_p"] = None
|
| 422 |
+
if "num_beams" not in gen_kwargs:
|
| 423 |
+
gen_kwargs["num_beams"] = 1
|
| 424 |
+
|
| 425 |
+
parts = self.split_text(prompt_question, ["<image>", "<speech>"])
|
| 426 |
+
input_ids = []
|
| 427 |
+
for part in parts:
|
| 428 |
+
if "<image>" == part:
|
| 429 |
+
input_ids += [IMAGE_TOKEN_INDEX]
|
| 430 |
+
elif "<speech>" == part:
|
| 431 |
+
input_ids += [SPEECH_TOKEN_INDEX]
|
| 432 |
+
else:
|
| 433 |
+
input_ids += self.tokenizer(part).input_ids
|
| 434 |
+
|
| 435 |
+
input_ids = torch.tensor(input_ids, dtype=torch.long).unsqueeze(0).to(self.device)
|
| 436 |
+
input_ids_list = [input_ids]
|
| 437 |
+
pad_token_ids = self.tokenizer.pad_token_id if self.tokenizer.pad_token_id is not None else self.tokenizer.eos_token_id
|
| 438 |
+
input_ids = self.pad_sequence(input_ids_list, batch_first=True, padding_value=pad_token_ids).to(self.device)
|
| 439 |
+
attention_masks = input_ids.ne(pad_token_ids).to(self.device)
|
| 440 |
+
input_ids = torch.tensor(input_ids, dtype=torch.long).squeeze(0).to(self.device)
|
| 441 |
+
if task_type == "image":
|
| 442 |
+
gen_kwargs["image_sizes"] = [batched_visuals[0][idx].size for idx in range(len(batched_visuals[0]))]
|
| 443 |
+
elif task_type == "video":
|
| 444 |
+
gen_kwargs["modalities"] = ["video"]
|
| 445 |
+
self._config.mm_spatial_pool_stride = self.mm_spatial_pool_stride
|
| 446 |
+
self._config.mm_spatial_pool_mode = self.mm_spatial_pool_mode
|
| 447 |
+
gen_kwargs["eos_token_id"] = self.tokenizer.eos_token_id
|
| 448 |
+
|
| 449 |
+
# These steps are not in LLaVA's original code, but are necessary for generation to work
|
| 450 |
+
# TODO: attention to this major generation step...
|
| 451 |
+
if "image_aspect_ratio" in gen_kwargs.keys():
|
| 452 |
+
gen_kwargs.pop("image_aspect_ratio")
|
| 453 |
+
try:
|
| 454 |
+
with torch.inference_mode():
|
| 455 |
+
cont = self.model.generate(input_ids, images=image_tensor, speech=speech, speech_lengths=speech_lengths, **gen_kwargs)
|
| 456 |
+
|
| 457 |
+
text_outputs = self.tokenizer.batch_decode(cont, skip_special_tokens=True)
|
| 458 |
+
except Exception as e:
|
| 459 |
+
raise e
|
| 460 |
+
|
| 461 |
+
text_outputs = [response.strip() for response in text_outputs]
|
| 462 |
+
res.extend(text_outputs)
|
| 463 |
+
self.cache_hook.add_partial("generate_until", (context, gen_kwargs), text_outputs)
|
| 464 |
+
pbar.update(1)
|
| 465 |
+
# reorder this group of results back to original unsorted form
|
| 466 |
+
res = re_ords.get_original(res)
|
| 467 |
+
|
| 468 |
+
pbar.close()
|
| 469 |
+
return res
|
| 470 |
+
|
| 471 |
+
def generate_until_multi_round(self, requests: List[Instance]) -> List[str]:
|
| 472 |
+
raise NotImplementedError("generate_until_multi_round is not implemented for EgoGPT")
|
LaViDa/eval/lmms_eval/models/minicpm_v.py
ADDED
|
@@ -0,0 +1,224 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import warnings
|
| 2 |
+
from typing import List, Optional, Tuple, Union
|
| 3 |
+
|
| 4 |
+
import torch
|
| 5 |
+
from accelerate import Accelerator, DistributedType
|
| 6 |
+
from accelerate.state import AcceleratorState
|
| 7 |
+
from tqdm import tqdm
|
| 8 |
+
from transformers import AutoModel, AutoTokenizer
|
| 9 |
+
|
| 10 |
+
from lmms_eval import utils
|
| 11 |
+
from lmms_eval.api.instance import Instance
|
| 12 |
+
from lmms_eval.api.model import lmms
|
| 13 |
+
from lmms_eval.api.registry import register_model
|
| 14 |
+
|
| 15 |
+
warnings.filterwarnings("ignore")
|
| 16 |
+
|
| 17 |
+
from loguru import logger as eval_logger
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
@register_model("minicpm_v")
|
| 21 |
+
class MiniCPM_V(lmms):
|
| 22 |
+
"""
|
| 23 |
+
MiniCPM_V Model
|
| 24 |
+
"""
|
| 25 |
+
|
| 26 |
+
def __init__(
|
| 27 |
+
self,
|
| 28 |
+
pretrained: str = "openbmb/MiniCPM-V",
|
| 29 |
+
device: Optional[str] = "cuda",
|
| 30 |
+
dtype: Optional[Union[str, torch.dtype]] = torch.bfloat16,
|
| 31 |
+
batch_size: Optional[Union[int, str]] = 1,
|
| 32 |
+
trust_remote_code: Optional[bool] = True,
|
| 33 |
+
**kwargs,
|
| 34 |
+
) -> None:
|
| 35 |
+
super().__init__()
|
| 36 |
+
# Do not use kwargs for now
|
| 37 |
+
assert kwargs == {}, f"Unexpected kwargs: {kwargs}"
|
| 38 |
+
|
| 39 |
+
accelerator = Accelerator()
|
| 40 |
+
if accelerator.num_processes > 1:
|
| 41 |
+
self._device = torch.device(f"cuda:{accelerator.local_process_index}")
|
| 42 |
+
else:
|
| 43 |
+
self._device = device
|
| 44 |
+
self._model = AutoModel.from_pretrained(pretrained, trust_remote_code=trust_remote_code, torch_dtype=dtype, device_map=self._device).to(dtype)
|
| 45 |
+
self._tokenizer = AutoTokenizer.from_pretrained(pretrained, trust_remote_code=trust_remote_code)
|
| 46 |
+
self._config = self._model.config
|
| 47 |
+
self.model.eval()
|
| 48 |
+
self.model.tie_weights()
|
| 49 |
+
self.batch_size_per_gpu = int(batch_size)
|
| 50 |
+
if accelerator.num_processes > 1:
|
| 51 |
+
assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported."
|
| 52 |
+
# If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model
|
| 53 |
+
# Also, you have to select zero stage 0 (equivalent to DDP) in order to make the prepare model works
|
| 54 |
+
# I tried to set different parameters in the kwargs to let default zero 2 stage works, but it didn't work.
|
| 55 |
+
if accelerator.distributed_type == DistributedType.DEEPSPEED:
|
| 56 |
+
kwargs = {
|
| 57 |
+
"train_micro_batch_size_per_gpu": self.batch_size_per_gpu,
|
| 58 |
+
"train_batch_size": self.batch_size_per_gpu * accelerator.num_processes,
|
| 59 |
+
}
|
| 60 |
+
AcceleratorState().deepspeed_plugin.deepspeed_config_process(must_match=True, **kwargs)
|
| 61 |
+
eval_logger.info("Detected that you are using DistributedType.DEEPSPEED. Make sure you run `accelerate config` and set zero stage to 0")
|
| 62 |
+
if accelerator.distributed_type == DistributedType.FSDP or accelerator.distributed_type == DistributedType.DEEPSPEED:
|
| 63 |
+
self._model = accelerator.prepare(self.model)
|
| 64 |
+
else:
|
| 65 |
+
self._model = accelerator.prepare_model(self.model, evaluation_mode=True)
|
| 66 |
+
self.accelerator = accelerator
|
| 67 |
+
if self.accelerator.is_local_main_process:
|
| 68 |
+
eval_logger.info(f"Using {accelerator.num_processes} devices with data parallelism")
|
| 69 |
+
self._rank = self.accelerator.local_process_index
|
| 70 |
+
self._world_size = self.accelerator.num_processes
|
| 71 |
+
else:
|
| 72 |
+
self.model.to(self._device)
|
| 73 |
+
self._rank = 0
|
| 74 |
+
self._world_size = 1
|
| 75 |
+
|
| 76 |
+
@property
|
| 77 |
+
def config(self):
|
| 78 |
+
# return the associated transformers.AutoConfig for the given pretrained model.
|
| 79 |
+
return self._config
|
| 80 |
+
|
| 81 |
+
@property
|
| 82 |
+
def tokenizer(self):
|
| 83 |
+
return self._tokenizer
|
| 84 |
+
|
| 85 |
+
@property
|
| 86 |
+
def model(self):
|
| 87 |
+
# returns the model, unwrapping it if using Accelerate
|
| 88 |
+
if hasattr(self, "accelerator"):
|
| 89 |
+
return self.accelerator.unwrap_model(self._model)
|
| 90 |
+
else:
|
| 91 |
+
return self._model
|
| 92 |
+
|
| 93 |
+
@property
|
| 94 |
+
def eot_token_id(self):
|
| 95 |
+
# we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
|
| 96 |
+
return self.tokenizer.eos_token_id
|
| 97 |
+
|
| 98 |
+
@property
|
| 99 |
+
def max_length(self):
|
| 100 |
+
return self._max_length
|
| 101 |
+
|
| 102 |
+
@property
|
| 103 |
+
def batch_size(self):
|
| 104 |
+
return self.batch_size_per_gpu
|
| 105 |
+
|
| 106 |
+
@property
|
| 107 |
+
def device(self):
|
| 108 |
+
return self._device
|
| 109 |
+
|
| 110 |
+
@property
|
| 111 |
+
def rank(self):
|
| 112 |
+
return self._rank
|
| 113 |
+
|
| 114 |
+
@property
|
| 115 |
+
def world_size(self):
|
| 116 |
+
return self._world_size
|
| 117 |
+
|
| 118 |
+
def tok_encode(self, string: str, left_truncate_len=None, add_special_tokens=None) -> List[int]:
|
| 119 |
+
""" """
|
| 120 |
+
add_special_tokens = False if add_special_tokens is None else add_special_tokens
|
| 121 |
+
encoding = self.tokenizer.encode(string, add_special_tokens=add_special_tokens)
|
| 122 |
+
# left-truncate the encoded context to be at most `left_truncate_len` tokens long
|
| 123 |
+
if left_truncate_len:
|
| 124 |
+
encoding = encoding[-left_truncate_len:]
|
| 125 |
+
return encoding
|
| 126 |
+
|
| 127 |
+
def tok_decode(self, tokens):
|
| 128 |
+
return self.tokenizer.decode(tokens)
|
| 129 |
+
|
| 130 |
+
def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
|
| 131 |
+
# TODO
|
| 132 |
+
assert False, "We have not implemented this function for MiniCPM_V yet"
|
| 133 |
+
|
| 134 |
+
def flatten(self, input):
|
| 135 |
+
new_list = []
|
| 136 |
+
for i in input:
|
| 137 |
+
for j in i:
|
| 138 |
+
new_list.append(j)
|
| 139 |
+
return new_list
|
| 140 |
+
|
| 141 |
+
def generate_until(self, requests: List[Instance]) -> List[str]:
|
| 142 |
+
res = []
|
| 143 |
+
|
| 144 |
+
def _collate(x):
|
| 145 |
+
# the negative sign on len(toks) sorts descending - this has a few advantages:
|
| 146 |
+
# - time estimates will always be over not underestimates, which is more useful for planning
|
| 147 |
+
# - to know the size of a batch when going through the list, you know the first one is always the batch
|
| 148 |
+
# padded context length. this is useful to simplify the batching logic and more importantly to make
|
| 149 |
+
# automatic adaptive batches much much easier to implement
|
| 150 |
+
# - any OOMs will happen right away rather than near the end
|
| 151 |
+
toks = self.tok_encode(x[0])
|
| 152 |
+
return -len(toks), x[0]
|
| 153 |
+
|
| 154 |
+
# we group requests by their generation_kwargs,
|
| 155 |
+
# so that we don't try to execute e.g. greedy sampling and temp=0.8 sampling
|
| 156 |
+
# in the same batch.
|
| 157 |
+
re_ords = utils.Collator([reg.args for reg in requests], _collate, grouping=True)
|
| 158 |
+
chunks = re_ords.get_batched(n=self.batch_size, batch_fn=None)
|
| 159 |
+
num_iters = len(requests) // self.batch_size if len(requests) % self.batch_size == 0 else len(requests) // self.batch_size + 1
|
| 160 |
+
pbar = tqdm(total=num_iters, disable=(self.rank != 0), desc="Model Responding")
|
| 161 |
+
for chunk in chunks:
|
| 162 |
+
contexts, all_gen_kwargs, doc_to_visual, doc_id, task, split = zip(*chunk)
|
| 163 |
+
task = task[0]
|
| 164 |
+
split = split[0]
|
| 165 |
+
visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]
|
| 166 |
+
visuals = self.flatten(visuals)
|
| 167 |
+
# we assume all gen kwargs in the batch are the same
|
| 168 |
+
# this is safe to assume because the `grouper` object ensures it.
|
| 169 |
+
gen_kwargs = all_gen_kwargs[0]
|
| 170 |
+
|
| 171 |
+
# Set default values for until and max_new_tokens
|
| 172 |
+
until = [self.tok_decode(self.eot_token_id)]
|
| 173 |
+
|
| 174 |
+
# Update values from gen_kwargs if present
|
| 175 |
+
if "until" in gen_kwargs:
|
| 176 |
+
until = gen_kwargs.pop("until")
|
| 177 |
+
if isinstance(until, str):
|
| 178 |
+
until = [until]
|
| 179 |
+
elif not isinstance(until, list):
|
| 180 |
+
raise ValueError(f"Expected `gen_kwargs['until']` to be of type Union[str,list] but got {type(until)}")
|
| 181 |
+
assert self.batch_size_per_gpu == 1, "Do not support batch_size_per_gpu > 1 for now"
|
| 182 |
+
assert len(visuals) == 1, "MiniCPM_V interface does not support bn_image > 1 for now"
|
| 183 |
+
context = contexts[0]
|
| 184 |
+
if "<image>" in context:
|
| 185 |
+
# minicpm does not expect the <image> tag
|
| 186 |
+
context = context.replace("<image>", "")
|
| 187 |
+
msgs = [{"role": "user", "content": context}]
|
| 188 |
+
|
| 189 |
+
gen_kwargs["image_sizes"] = [visuals[idx].size for idx in range(len(visuals))]
|
| 190 |
+
if "max_new_tokens" not in gen_kwargs:
|
| 191 |
+
gen_kwargs["max_new_tokens"] = 1024
|
| 192 |
+
if "temperature" not in gen_kwargs:
|
| 193 |
+
gen_kwargs["temperature"] = 0
|
| 194 |
+
if "top_p" not in gen_kwargs:
|
| 195 |
+
gen_kwargs["top_p"] = None
|
| 196 |
+
if "num_beams" not in gen_kwargs:
|
| 197 |
+
gen_kwargs["num_beams"] = 1
|
| 198 |
+
try:
|
| 199 |
+
# ominicpm does not give much information on how they do eval so I just use the chat format.
|
| 200 |
+
response, context, _ = self.model.chat(
|
| 201 |
+
image=visuals[0],
|
| 202 |
+
msgs=msgs,
|
| 203 |
+
context=None,
|
| 204 |
+
tokenizer=self.tokenizer,
|
| 205 |
+
sampling=True if gen_kwargs["temperature"] > 0 else False,
|
| 206 |
+
temperature=gen_kwargs["temperature"],
|
| 207 |
+
top_p=gen_kwargs["top_p"],
|
| 208 |
+
num_beams=gen_kwargs["num_beams"],
|
| 209 |
+
max_new_tokens=gen_kwargs["max_new_tokens"],
|
| 210 |
+
)
|
| 211 |
+
except Exception as e:
|
| 212 |
+
eval_logger.error(f"Error {e} in generating")
|
| 213 |
+
cont = ""
|
| 214 |
+
res.append(response)
|
| 215 |
+
self.cache_hook.add_partial("generate_until", (context, gen_kwargs), response)
|
| 216 |
+
pbar.update(1)
|
| 217 |
+
# reorder this group of results back to original unsorted form
|
| 218 |
+
res = re_ords.get_original(res)
|
| 219 |
+
|
| 220 |
+
pbar.close()
|
| 221 |
+
return res
|
| 222 |
+
|
| 223 |
+
def generate_until_multi_round(self, requests) -> List[str]:
|
| 224 |
+
raise NotImplementedError("TODO: Implement multi-round generation")
|
LaViDa/eval/lmms_eval/models/moviechat.py
ADDED
|
@@ -0,0 +1,452 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import copy
|
| 2 |
+
import json
|
| 3 |
+
import logging
|
| 4 |
+
import math
|
| 5 |
+
import os
|
| 6 |
+
import os.path as osp
|
| 7 |
+
import queue
|
| 8 |
+
import re
|
| 9 |
+
import warnings
|
| 10 |
+
from datetime import timedelta
|
| 11 |
+
from typing import List, Optional, Tuple, Union
|
| 12 |
+
|
| 13 |
+
import av
|
| 14 |
+
import einops
|
| 15 |
+
import numpy as np
|
| 16 |
+
import PIL
|
| 17 |
+
import torch
|
| 18 |
+
import transformers
|
| 19 |
+
from accelerate import Accelerator, DistributedType, InitProcessGroupKwargs
|
| 20 |
+
from accelerate.state import AcceleratorState
|
| 21 |
+
from decord import VideoReader, cpu
|
| 22 |
+
from huggingface_hub import snapshot_download
|
| 23 |
+
from moviepy.video.io.VideoFileClip import VideoFileClip
|
| 24 |
+
from packaging import version
|
| 25 |
+
from PIL import Image
|
| 26 |
+
from scipy.spatial.distance import cosine
|
| 27 |
+
from skimage import transform
|
| 28 |
+
from torchvision import transforms
|
| 29 |
+
from torchvision.transforms.functional import InterpolationMode
|
| 30 |
+
from tqdm import tqdm
|
| 31 |
+
from transformers import StoppingCriteria, StoppingCriteriaList
|
| 32 |
+
|
| 33 |
+
from lmms_eval import utils
|
| 34 |
+
from lmms_eval.api.instance import Instance
|
| 35 |
+
from lmms_eval.api.model import lmms
|
| 36 |
+
from lmms_eval.api.registry import register_model
|
| 37 |
+
|
| 38 |
+
# Suppress warnings
|
| 39 |
+
warnings.filterwarnings("ignore")
|
| 40 |
+
|
| 41 |
+
# Configure logging
|
| 42 |
+
eval_logger = logging.getLogger("lmms-eval")
|
| 43 |
+
|
| 44 |
+
# Enable TF32 for CUDA
|
| 45 |
+
torch.backends.cuda.matmul.allow_tf32 = True
|
| 46 |
+
|
| 47 |
+
# Import LLaVA modules
|
| 48 |
+
try:
|
| 49 |
+
from MovieChat.common.registry import registry
|
| 50 |
+
except ImportError as e:
|
| 51 |
+
eval_logger.debug(
|
| 52 |
+
f"MovieChat is not installed. First, install MovieChat by 'https://github.com/rese1f/MovieChat.git' and 'cd MovieChat'. Change the torch version with `python -m pip install torch==2.0.1 torchvision==0.15.2 --extra-index-url https://download.pytorch.org/whl/cu118`"
|
| 53 |
+
)
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
# Determine best attention implementation
|
| 57 |
+
if version.parse(torch.__version__) >= version.parse("2.1.2"):
|
| 58 |
+
best_fit_attn_implementation = "sdpa"
|
| 59 |
+
else:
|
| 60 |
+
best_fit_attn_implementation = "eager"
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
class StoppingCriteriaSub(StoppingCriteria):
|
| 64 |
+
def __init__(self, stops=[], encounters=1):
|
| 65 |
+
super().__init__()
|
| 66 |
+
self.stops = stops
|
| 67 |
+
|
| 68 |
+
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor):
|
| 69 |
+
for stop in self.stops:
|
| 70 |
+
if torch.all((stop == input_ids[0][-len(stop) :])).item():
|
| 71 |
+
return True
|
| 72 |
+
|
| 73 |
+
return False
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
@register_model("moviechat")
|
| 77 |
+
class MovieChat(lmms):
|
| 78 |
+
"""
|
| 79 |
+
MovieChat Model
|
| 80 |
+
"""
|
| 81 |
+
|
| 82 |
+
def __init__(
|
| 83 |
+
self,
|
| 84 |
+
truncation: Optional[bool] = True,
|
| 85 |
+
device: Optional[str] = "cuda:0",
|
| 86 |
+
batch_size: Optional[Union[int, str]] = 1,
|
| 87 |
+
pretrained_llama_model: str = "Enxin/MovieChat-vicuna",
|
| 88 |
+
pretrained_llama_proj_model: str = "Enxin/MovieChat-proj",
|
| 89 |
+
attn_implementation: Optional[str] = best_fit_attn_implementation,
|
| 90 |
+
device_map: Optional[str] = "cuda:0",
|
| 91 |
+
use_cache: Optional[bool] = True,
|
| 92 |
+
truncate_context: Optional[bool] = False, # whether to truncate the context in generation, set it False for LLaVA-1.6
|
| 93 |
+
customized_config: Optional[str] = None, # ends in json
|
| 94 |
+
short_memory_length: Optional[int] = 18,
|
| 95 |
+
long_memory_length: Optional[int] = 256,
|
| 96 |
+
sliding_window_length: Optional[int] = 8,
|
| 97 |
+
merge_frame_length: Optional[int] = 2,
|
| 98 |
+
tmp_folder: Optional[str] = "tmp/",
|
| 99 |
+
**kwargs,
|
| 100 |
+
) -> None:
|
| 101 |
+
super().__init__()
|
| 102 |
+
# Do not use kwargs for now
|
| 103 |
+
assert kwargs == {}, f"Unexpected kwargs: {kwargs}"
|
| 104 |
+
|
| 105 |
+
accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52))
|
| 106 |
+
accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs])
|
| 107 |
+
if accelerator.num_processes > 1:
|
| 108 |
+
self._device = torch.device(f"cuda:{accelerator.local_process_index}")
|
| 109 |
+
self.device_map = f"cuda:{accelerator.local_process_index}"
|
| 110 |
+
elif accelerator.num_processes == 1 and device_map == "auto":
|
| 111 |
+
self._device = torch.device(device)
|
| 112 |
+
self.device_map = device_map
|
| 113 |
+
else:
|
| 114 |
+
self._device = torch.device(f"cuda:{accelerator.local_process_index}")
|
| 115 |
+
self.device_map = f"cuda:{accelerator.local_process_index}"
|
| 116 |
+
|
| 117 |
+
llama_model = snapshot_download(repo_id=pretrained_llama_model) if not osp.isdir(pretrained_llama_model) else pretrained_llama_model
|
| 118 |
+
llama_proj_pth = snapshot_download(repo_id=pretrained_llama_proj_model) if not osp.isdir(pretrained_llama_proj_model) else pretrained_llama_proj_model
|
| 119 |
+
llama_proj = osp.join(llama_proj_pth, "finetune-vicuna7b-v2.pth")
|
| 120 |
+
model_config = {
|
| 121 |
+
"arch": "moviechat",
|
| 122 |
+
"model_type": "pretrain_vicuna",
|
| 123 |
+
"freeze_vit": True,
|
| 124 |
+
"freeze_qformer": True,
|
| 125 |
+
"max_txt_len": 256,
|
| 126 |
+
"end_sym": "###",
|
| 127 |
+
"low_resource": False,
|
| 128 |
+
"frozen_llama_proj": False,
|
| 129 |
+
"llama_model": llama_model,
|
| 130 |
+
"llama_proj_model": llama_proj,
|
| 131 |
+
}
|
| 132 |
+
|
| 133 |
+
model_cls = registry.get_model_class(model_config["arch"])
|
| 134 |
+
self._model = model_cls.from_config(model_config).to(self.device_map)
|
| 135 |
+
|
| 136 |
+
vis_processor_cfg = {
|
| 137 |
+
"name": "alpro_video_eval",
|
| 138 |
+
"n_frms": 8,
|
| 139 |
+
}
|
| 140 |
+
self.transform = transforms.Compose(
|
| 141 |
+
[transforms.Resize((224, 224)), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])] # Resize to 224x224 # Convert PIL Image to Tensor with shape [C, H, W] # Normalize
|
| 142 |
+
)
|
| 143 |
+
self._image_processor = registry.get_processor_class(vis_processor_cfg["name"]).from_config(vis_processor_cfg)
|
| 144 |
+
|
| 145 |
+
self.model.short_memory_length = short_memory_length
|
| 146 |
+
self.model.long_memory_length = long_memory_length
|
| 147 |
+
self.merge_frame_length = merge_frame_length
|
| 148 |
+
self.sliding_window_length = sliding_window_length
|
| 149 |
+
self.num_clips = (self.model.long_memory_length // self.merge_frame_length) * ((self.model.short_memory_length - self.merge_frame_length) // self.sliding_window_length)
|
| 150 |
+
self.tmp_folder = tmp_folder
|
| 151 |
+
|
| 152 |
+
self._tokenizer = self.model.llama_tokenizer
|
| 153 |
+
stop_words_ids = [torch.tensor([835]).to(self.device), torch.tensor([2277, 29937]).to(self.device)] # '###' can be encoded in two different ways.
|
| 154 |
+
self.stopping_criteria = StoppingCriteriaList([StoppingCriteriaSub(stops=stop_words_ids)])
|
| 155 |
+
|
| 156 |
+
self.model.eval()
|
| 157 |
+
self.truncation = truncation
|
| 158 |
+
self.batch_size_per_gpu = int(batch_size)
|
| 159 |
+
self.use_cache = use_cache
|
| 160 |
+
self.truncate_context = truncate_context
|
| 161 |
+
assert self.batch_size_per_gpu == 1, "MovieChat currently does not support batched generation."
|
| 162 |
+
|
| 163 |
+
if accelerator.num_processes > 1:
|
| 164 |
+
assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported."
|
| 165 |
+
# If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model
|
| 166 |
+
# Also, you have to select zero stage 0 (equivalent to DDP) in order to make the prepare model works
|
| 167 |
+
# I tried to set different parameters in the kwargs to let default zero 2 stage works, but it didn't work.
|
| 168 |
+
if accelerator.distributed_type == DistributedType.DEEPSPEED:
|
| 169 |
+
kwargs = {
|
| 170 |
+
"train_micro_batch_size_per_gpu": self.batch_size_per_gpu,
|
| 171 |
+
"train_batch_size": self.batch_size_per_gpu * accelerator.num_processes,
|
| 172 |
+
}
|
| 173 |
+
AcceleratorState().deepspeed_plugin.deepspeed_config_process(must_match=True, **kwargs)
|
| 174 |
+
eval_logger.info("Detected that you are using DistributedType.DEEPSPEED. Make sure you run `accelerate config` and set zero stage to 0")
|
| 175 |
+
|
| 176 |
+
if accelerator.distributed_type == DistributedType.FSDP or accelerator.distributed_type == DistributedType.DEEPSPEED:
|
| 177 |
+
self._model = accelerator.prepare(self.model)
|
| 178 |
+
else:
|
| 179 |
+
self._model = accelerator.prepare_model(self.model, evaluation_mode=True)
|
| 180 |
+
self.accelerator = accelerator
|
| 181 |
+
if self.accelerator.is_local_main_process:
|
| 182 |
+
eval_logger.info(f"Using {accelerator.num_processes} devices with data parallelism")
|
| 183 |
+
self._rank = self.accelerator.local_process_index
|
| 184 |
+
self._world_size = self.accelerator.num_processes
|
| 185 |
+
|
| 186 |
+
elif accelerator.num_processes == 1 and device_map == "auto":
|
| 187 |
+
eval_logger.info(f"Using {accelerator.num_processes} devices with tensor parallelism")
|
| 188 |
+
self._rank = 0
|
| 189 |
+
self._world_size = 1
|
| 190 |
+
|
| 191 |
+
else:
|
| 192 |
+
eval_logger.info(f"Using single device: {self._device}")
|
| 193 |
+
self.model.to(self._device)
|
| 194 |
+
self._rank = 0
|
| 195 |
+
self._world_size = 1
|
| 196 |
+
|
| 197 |
+
@property
|
| 198 |
+
def config(self):
|
| 199 |
+
# return the associated transformers.AutoConfig for the given pretrained model.
|
| 200 |
+
return self._config
|
| 201 |
+
|
| 202 |
+
@property
|
| 203 |
+
def tokenizer(self):
|
| 204 |
+
return self._tokenizer
|
| 205 |
+
|
| 206 |
+
@property
|
| 207 |
+
def model(self):
|
| 208 |
+
# returns the model, unwrapping it if using Accelerate
|
| 209 |
+
if hasattr(self, "accelerator"):
|
| 210 |
+
return self.accelerator.unwrap_model(self._model)
|
| 211 |
+
else:
|
| 212 |
+
return self._model
|
| 213 |
+
|
| 214 |
+
@property
|
| 215 |
+
def eot_token_id(self):
|
| 216 |
+
# we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
|
| 217 |
+
return self.tokenizer.eos_token_id
|
| 218 |
+
|
| 219 |
+
@property
|
| 220 |
+
def max_length(self):
|
| 221 |
+
return self._max_length
|
| 222 |
+
|
| 223 |
+
@property
|
| 224 |
+
def batch_size(self):
|
| 225 |
+
return self.batch_size_per_gpu
|
| 226 |
+
|
| 227 |
+
@property
|
| 228 |
+
def device(self):
|
| 229 |
+
return self._device
|
| 230 |
+
|
| 231 |
+
@property
|
| 232 |
+
def rank(self):
|
| 233 |
+
return self._rank
|
| 234 |
+
|
| 235 |
+
@property
|
| 236 |
+
def world_size(self):
|
| 237 |
+
return self._world_size
|
| 238 |
+
|
| 239 |
+
def tok_encode(self, string: str, left_truncate_len=None, add_special_tokens=None) -> List[int]:
|
| 240 |
+
""" """
|
| 241 |
+
add_special_tokens = False if add_special_tokens is None else add_special_tokens
|
| 242 |
+
encoding = self.tokenizer.encode(string, add_special_tokens=add_special_tokens)
|
| 243 |
+
# left-truncate the encoded context to be at most `left_truncate_len` tokens long
|
| 244 |
+
if left_truncate_len:
|
| 245 |
+
encoding = encoding[-left_truncate_len:]
|
| 246 |
+
return encoding
|
| 247 |
+
|
| 248 |
+
def tok_decode(self, tokens):
|
| 249 |
+
try:
|
| 250 |
+
return self.tokenizer.decode(tokens)
|
| 251 |
+
except:
|
| 252 |
+
return self.tokenizer.decode([tokens])
|
| 253 |
+
|
| 254 |
+
def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
|
| 255 |
+
# TODO
|
| 256 |
+
raise NotImplementedError("MovieChat only supports generation.")
|
| 257 |
+
|
| 258 |
+
def flatten(self, input):
|
| 259 |
+
new_list = []
|
| 260 |
+
for i in input:
|
| 261 |
+
for j in i:
|
| 262 |
+
new_list.append(j)
|
| 263 |
+
return new_list
|
| 264 |
+
|
| 265 |
+
def get_context_emb(self, input_text, img_list):
|
| 266 |
+
prompt_1 = "You are able to understand the visual content that the user provides.Follow the instructions carefully and explain your answers in details.###Human: <Video><ImageHere></Video>"
|
| 267 |
+
prompt_2 = input_text
|
| 268 |
+
prompt_3 = "###Assistant:"
|
| 269 |
+
|
| 270 |
+
prompt = prompt_1 + " " + prompt_2 + prompt_3
|
| 271 |
+
|
| 272 |
+
prompt_segs = prompt.split("<ImageHere>")
|
| 273 |
+
assert len(prompt_segs) == len(img_list) + 1, "Unmatched numbers of image placeholders and images."
|
| 274 |
+
seg_tokens = [
|
| 275 |
+
self.model.llama_tokenizer(seg, return_tensors="pt", add_special_tokens=i == 0).to(self.device).input_ids
|
| 276 |
+
# only add bos to the first seg
|
| 277 |
+
for i, seg in enumerate(prompt_segs)
|
| 278 |
+
]
|
| 279 |
+
seg_embs = [self.model.llama_model.model.embed_tokens(seg_t) for seg_t in seg_tokens]
|
| 280 |
+
|
| 281 |
+
mixed_embs = [emb for pair in zip(seg_embs[:-1], img_list) for emb in pair] + [seg_embs[-1]]
|
| 282 |
+
mixed_embs = torch.cat(mixed_embs, dim=1)
|
| 283 |
+
return mixed_embs
|
| 284 |
+
|
| 285 |
+
def answer(self, img_list, input_text, max_new_tokens=300, num_beams=1, min_length=1, top_p=0.9, repetition_penalty=1.0, length_penalty=1, temperature=1.0, max_length=2000):
|
| 286 |
+
embs = self.get_context_emb(input_text, img_list)
|
| 287 |
+
|
| 288 |
+
current_max_len = embs.shape[1] + max_new_tokens
|
| 289 |
+
if current_max_len - max_length > 0:
|
| 290 |
+
print("Warning: The number of tokens in current conversation exceeds the max length. " "The model will not see the contexts outside the range.")
|
| 291 |
+
begin_idx = max(0, current_max_len - max_length)
|
| 292 |
+
|
| 293 |
+
embs = embs[:, begin_idx:]
|
| 294 |
+
|
| 295 |
+
outputs = self.model.llama_model.generate(
|
| 296 |
+
inputs_embeds=embs,
|
| 297 |
+
max_new_tokens=max_new_tokens,
|
| 298 |
+
stopping_criteria=self.stopping_criteria,
|
| 299 |
+
num_beams=num_beams,
|
| 300 |
+
do_sample=True,
|
| 301 |
+
min_length=min_length,
|
| 302 |
+
top_p=top_p,
|
| 303 |
+
repetition_penalty=repetition_penalty,
|
| 304 |
+
length_penalty=length_penalty,
|
| 305 |
+
temperature=temperature,
|
| 306 |
+
)
|
| 307 |
+
|
| 308 |
+
output_token = outputs[0]
|
| 309 |
+
if output_token[0] == 0: # the model might output a unknow token
|
| 310 |
+
output_token = output_token[1:]
|
| 311 |
+
if output_token[0] == 1: # some users find that there is a start token at the beginning. remove it
|
| 312 |
+
output_token = output_token[1:]
|
| 313 |
+
output_text = self.model.llama_tokenizer.decode(output_token, add_special_tokens=False)
|
| 314 |
+
output_text = output_text.split("###")[0] # remove the stop sign '###'
|
| 315 |
+
output_text = output_text.split("Assistant:")[-1].strip()
|
| 316 |
+
return output_text, output_token.cpu().numpy()
|
| 317 |
+
|
| 318 |
+
def generate_until(self, requests: List[Instance]) -> List[str]:
|
| 319 |
+
res = []
|
| 320 |
+
|
| 321 |
+
def _collate(x):
|
| 322 |
+
# the negative sign on len(toks) sorts descending - this has a few advantages:
|
| 323 |
+
# - time estimates will always be over not underestimates, which is more useful for planning
|
| 324 |
+
# - to know the size of a batch when going through the list, you know the first one is always the batch
|
| 325 |
+
# padded context length. this is useful to simplify the batching logic and more importantly to make
|
| 326 |
+
# automatic adaptive batches much much easier to implement
|
| 327 |
+
# - any OOMs will happen right away rather than near the end
|
| 328 |
+
toks = self.tok_encode(x[0])
|
| 329 |
+
return -len(toks), x[0]
|
| 330 |
+
|
| 331 |
+
# we group requests by their generation_kwargs,
|
| 332 |
+
# so that we don't try to execute e.g. greedy sampling and temp=0.8 sampling
|
| 333 |
+
# in the same batch.
|
| 334 |
+
metadata = requests[0].metadata
|
| 335 |
+
re_ords = utils.Collator([reg.args for reg in requests], _collate, grouping=True)
|
| 336 |
+
chunks = re_ords.get_batched(n=self.batch_size, batch_fn=None)
|
| 337 |
+
num_iters = len(requests) // self.batch_size if len(requests) % self.batch_size == 0 else len(requests) // self.batch_size + 1
|
| 338 |
+
pbar = tqdm(total=num_iters, disable=(self.rank != 0), desc="Model Responding")
|
| 339 |
+
for chunk in chunks:
|
| 340 |
+
batched_contexts, all_gen_kwargs, batched_doc_to_visual, batched_doc_id, batched_task, batched_split = zip(*chunk)
|
| 341 |
+
task = batched_task[0]
|
| 342 |
+
split = batched_split[0]
|
| 343 |
+
batched_visuals = [batched_doc_to_visual[0](self.task_dict[task][split][ids]) for ids in batched_doc_id] # [B, N]
|
| 344 |
+
assert len(batched_visuals) == 1
|
| 345 |
+
|
| 346 |
+
# we assume all gen kwargs in the batch are the same
|
| 347 |
+
# this is safe to assume because the `grouper` object ensures it.
|
| 348 |
+
gen_kwargs = all_gen_kwargs[0]
|
| 349 |
+
if "until" in gen_kwargs:
|
| 350 |
+
gen_kwargs.pop("until")
|
| 351 |
+
|
| 352 |
+
text_outputs = []
|
| 353 |
+
|
| 354 |
+
for visual, context in zip(batched_visuals, batched_contexts):
|
| 355 |
+
if type(visual[0]) == PIL.Image.Image and "task_type" not in metadata and "sample_frames" not in metadata: # For image task
|
| 356 |
+
raise NotImplementedError("MovieChat only supports video inputs.")
|
| 357 |
+
|
| 358 |
+
elif "task_type" in metadata and metadata["task_type"] == "video" and "sample_frames" in metadata:
|
| 359 |
+
raise NotImplementedError("MovieChat only supports video inputs.")
|
| 360 |
+
|
| 361 |
+
elif type(visual[0]) == str: # For video task
|
| 362 |
+
image_tensor = []
|
| 363 |
+
self.model.short_memory_buffer = []
|
| 364 |
+
self.model.long_memory_buffer = []
|
| 365 |
+
img_list = []
|
| 366 |
+
# try:
|
| 367 |
+
os.makedirs(self.tmp_folder, exist_ok=True)
|
| 368 |
+
|
| 369 |
+
video = VideoFileClip(visual[0])
|
| 370 |
+
clip_duration = video.duration / self.num_clips
|
| 371 |
+
|
| 372 |
+
cur_frame = 0
|
| 373 |
+
for i in range(self.num_clips):
|
| 374 |
+
preprocess_frames = []
|
| 375 |
+
start_time = i * clip_duration
|
| 376 |
+
end_time = start_time + clip_duration
|
| 377 |
+
# uniformly sample self.sliding_window_length frames from the video from start_time to end_time
|
| 378 |
+
frames = list(video.subclip(start_time, end_time).iter_frames(fps=self.sliding_window_length / clip_duration))[: self.sliding_window_length]
|
| 379 |
+
for frame in frames:
|
| 380 |
+
frame = Image.fromarray(frame)
|
| 381 |
+
frame_tensor = self.transform(frame)
|
| 382 |
+
frame_tensor = frame_tensor.permute(2, 0, 1)
|
| 383 |
+
frame_tensor = frame_tensor.unsqueeze(0)
|
| 384 |
+
frame_tensor = self._image_processor.transform(frame_tensor)
|
| 385 |
+
frame_tensor = frame_tensor.squeeze(-1).permute(1, 2, 0)
|
| 386 |
+
preprocess_frames.append(frame_tensor)
|
| 387 |
+
|
| 388 |
+
frames_tensor = torch.stack(preprocess_frames, dim=0)
|
| 389 |
+
|
| 390 |
+
image_embeds = self.model.ln_vision(self.model.visual_encoder(frames_tensor.half().to(self.device)))
|
| 391 |
+
image_atts = torch.ones(image_embeds.size()[:-1], dtype=torch.long).to(self.device)
|
| 392 |
+
query_tokens = self.model.query_tokens.expand(image_embeds.shape[0], -1, -1)
|
| 393 |
+
query_output = self.model.Qformer.bert(
|
| 394 |
+
query_embeds=query_tokens,
|
| 395 |
+
encoder_hidden_states=image_embeds,
|
| 396 |
+
encoder_attention_mask=image_atts,
|
| 397 |
+
return_dict=True,
|
| 398 |
+
)
|
| 399 |
+
encoded_window = query_output.last_hidden_state
|
| 400 |
+
|
| 401 |
+
for frame in encoded_window:
|
| 402 |
+
if cur_frame < (self.model.short_memory_length - self.merge_frame_length):
|
| 403 |
+
if len(self.model.short_memory_buffer) == self.model.short_memory_length:
|
| 404 |
+
self.model.short_memory_buffer.pop(0)
|
| 405 |
+
self.model.short_memory_buffer.append(frame)
|
| 406 |
+
cur_frame += 1
|
| 407 |
+
|
| 408 |
+
if cur_frame == (self.model.short_memory_length - self.merge_frame_length):
|
| 409 |
+
cur_frame = 0
|
| 410 |
+
|
| 411 |
+
# merge short_memory_frames
|
| 412 |
+
similar_list = []
|
| 413 |
+
for frame_i in range(len(self.model.short_memory_buffer) - 1):
|
| 414 |
+
scores = self.model.short_memory_buffer[frame_i] @ self.model.short_memory_buffer[frame_i + 1].transpose(-1, -2)
|
| 415 |
+
frame_silimar = torch.mean(scores)
|
| 416 |
+
similar_list.append(frame_silimar)
|
| 417 |
+
|
| 418 |
+
while len(self.model.short_memory_buffer) > self.merge_frame_length:
|
| 419 |
+
max_value = max(similar_list)
|
| 420 |
+
max_index = similar_list.index(max_value)
|
| 421 |
+
new_frame_feature = (self.model.short_memory_buffer[max_index].cpu() + self.model.short_memory_buffer[max_index + 1].cpu()) / 2
|
| 422 |
+
self.model.short_memory_buffer[max_index] = new_frame_feature.cuda()
|
| 423 |
+
del self.model.short_memory_buffer[max_index + 1]
|
| 424 |
+
similar_list = []
|
| 425 |
+
for frame_i in range(len(self.model.short_memory_buffer) - 1):
|
| 426 |
+
scores = self.model.short_memory_buffer[frame_i] @ self.model.short_memory_buffer[frame_i + 1].transpose(-1, -2)
|
| 427 |
+
frame_silimar = torch.mean(scores)
|
| 428 |
+
similar_list.append(frame_silimar)
|
| 429 |
+
|
| 430 |
+
for frame in self.model.short_memory_buffer:
|
| 431 |
+
self.model.long_memory_buffer.append(frame)
|
| 432 |
+
|
| 433 |
+
cur_image = self.model.encode_image(preprocess_frames[-1].unsqueeze(0).unsqueeze(2).half(), self.device)
|
| 434 |
+
video_emb, _ = self.model.encode_long_video(cur_image, device=self.device, middle_video=False)
|
| 435 |
+
img_list.append(video_emb)
|
| 436 |
+
llm_message = self.answer(img_list=img_list, input_text=context, num_beams=1, temperature=1.0, max_new_tokens=300, max_length=2000)[0]
|
| 437 |
+
text_outputs.append(llm_message)
|
| 438 |
+
|
| 439 |
+
# except Exception as e:
|
| 440 |
+
# eval_logger.error(f"Error {e} in loading video")
|
| 441 |
+
# image_tensor = None
|
| 442 |
+
|
| 443 |
+
text_outputs = [response.strip() for response in text_outputs]
|
| 444 |
+
res.extend(text_outputs)
|
| 445 |
+
print(text_outputs)
|
| 446 |
+
self.cache_hook.add_partial("generate_until", (context, gen_kwargs), text_outputs)
|
| 447 |
+
pbar.update(1)
|
| 448 |
+
# reorder this group of results back to original unsorted form
|
| 449 |
+
res = re_ords.get_original(res)
|
| 450 |
+
|
| 451 |
+
pbar.close()
|
| 452 |
+
return res
|
LaViDa/eval/lmms_eval/models/video_llava.py
ADDED
|
@@ -0,0 +1,214 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import math
|
| 2 |
+
from datetime import timedelta
|
| 3 |
+
from typing import List, Optional, Tuple, Union
|
| 4 |
+
|
| 5 |
+
import numpy as np
|
| 6 |
+
import torch
|
| 7 |
+
from accelerate import Accelerator, DistributedType, InitProcessGroupKwargs
|
| 8 |
+
from accelerate.state import AcceleratorState
|
| 9 |
+
from loguru import logger
|
| 10 |
+
from tqdm import tqdm
|
| 11 |
+
from transformers import AutoConfig
|
| 12 |
+
|
| 13 |
+
from lmms_eval import utils
|
| 14 |
+
from lmms_eval.api.instance import Instance
|
| 15 |
+
from lmms_eval.api.model import lmms
|
| 16 |
+
from lmms_eval.api.registry import register_model
|
| 17 |
+
from lmms_eval.utils import stop_sequences_criteria
|
| 18 |
+
|
| 19 |
+
eval_logger = logger
|
| 20 |
+
|
| 21 |
+
from transformers import VideoLlavaForConditionalGeneration, VideoLlavaProcessor
|
| 22 |
+
|
| 23 |
+
from lmms_eval.models.model_utils.load_video import read_video_pyav
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
@register_model("video_llava")
|
| 27 |
+
class VideoLLaVA(lmms):
|
| 28 |
+
def __init__(
|
| 29 |
+
self,
|
| 30 |
+
pretrained: str = "LanguageBind/Video-LLaVA-7B-hf",
|
| 31 |
+
truncation: Optional[bool] = True,
|
| 32 |
+
device: Optional[str] = "cuda:0",
|
| 33 |
+
dtype: Optional[Union[str, torch.dtype]] = "auto",
|
| 34 |
+
batch_size: Optional[Union[int, str]] = 1,
|
| 35 |
+
trust_remote_code: Optional[bool] = False,
|
| 36 |
+
revision=None,
|
| 37 |
+
attn_implementation=(
|
| 38 |
+
"sdpa" if torch.__version__ > "2.1.2" else "eager"
|
| 39 |
+
), # inference implementation for attention, can be "sdpa", "eager", "flash_attention_2". Seems FA2 is not effective during inference: https://discuss.huggingface.co/t/flash-attention-has-no-effect-on-inference/73453/5
|
| 40 |
+
device_map="cuda:0",
|
| 41 |
+
conv_template="llava_v1",
|
| 42 |
+
use_cache=True,
|
| 43 |
+
truncate_context=False,
|
| 44 |
+
num_frames: int = 8, # whether to truncate the context in generation, set it False for LLaVA-1.6
|
| 45 |
+
**kwargs,
|
| 46 |
+
) -> None:
|
| 47 |
+
super().__init__()
|
| 48 |
+
accelerator_kwargs = InitProcessGroupKwargs(timeout=timedelta(weeks=52))
|
| 49 |
+
accelerator = Accelerator(kwargs_handlers=[accelerator_kwargs])
|
| 50 |
+
if accelerator.num_processes > 1:
|
| 51 |
+
self._device = torch.device(f"cuda:{accelerator.local_process_index}")
|
| 52 |
+
self.device_map = f"cuda:{accelerator.local_process_index}"
|
| 53 |
+
elif accelerator.num_processes == 1 and device_map == "auto":
|
| 54 |
+
self._device = torch.device(device)
|
| 55 |
+
self.device_map = device_map
|
| 56 |
+
else:
|
| 57 |
+
self._device = torch.device(f"cuda:{accelerator.local_process_index}")
|
| 58 |
+
self.device_map = f"cuda:{accelerator.local_process_index}"
|
| 59 |
+
|
| 60 |
+
self.pretrained = pretrained
|
| 61 |
+
self._model = VideoLlavaForConditionalGeneration.from_pretrained(pretrained)
|
| 62 |
+
self._processor = VideoLlavaProcessor.from_pretrained(pretrained)
|
| 63 |
+
self.prompt = "USER: <video>{}? ASSISTANT:"
|
| 64 |
+
self.num_frames = num_frames
|
| 65 |
+
assert num_frames == 8, "num_frames must be 8 https://github.com/huggingface/transformers/blob/bdb9106f247fca48a71eb384be25dbbd29b065a8/src/transformers/models/video_llava/modeling_video_llava.py#L379"
|
| 66 |
+
# self.model_name = get_model_name_from_path(pretrained)
|
| 67 |
+
# self._tokenizer, self._model, self.processor, self._max_length = load_pretrained_model(pretrained, None, self.model_name, device_map=self.device_map)
|
| 68 |
+
# self.video_processor = self.processor["video"]
|
| 69 |
+
self._config = self._model.config
|
| 70 |
+
self.model.eval()
|
| 71 |
+
self.model.tie_weights()
|
| 72 |
+
self.truncation = truncation
|
| 73 |
+
self.batch_size_per_gpu = int(batch_size)
|
| 74 |
+
self.conv_template = conv_template
|
| 75 |
+
self.use_cache = use_cache
|
| 76 |
+
self.truncate_context = truncate_context
|
| 77 |
+
# assert self.batch_size_per_gpu == 1, "Llava currently does not support batched generation. See https://github.com/haotian-liu/LLaVA/issues/754. HF Llava also has this issue."
|
| 78 |
+
if accelerator.num_processes > 1:
|
| 79 |
+
assert accelerator.distributed_type in [DistributedType.FSDP, DistributedType.MULTI_GPU, DistributedType.DEEPSPEED], "Unsupported distributed type provided. Only DDP and FSDP are supported."
|
| 80 |
+
# If you want to use DistributedType.DEEPSPEED, you have to run accelerate config before using the model
|
| 81 |
+
# Also, you have to select zero stage 0 (equivalent to DDP) in order to make the prepare model works
|
| 82 |
+
# I tried to set different parameters in the kwargs to let default zero 2 stage works, but it didn't work.
|
| 83 |
+
if accelerator.distributed_type == DistributedType.DEEPSPEED:
|
| 84 |
+
kwargs = {
|
| 85 |
+
"train_micro_batch_size_per_gpu": self.batch_size_per_gpu,
|
| 86 |
+
"train_batch_size": self.batch_size_per_gpu * accelerator.num_processes,
|
| 87 |
+
}
|
| 88 |
+
AcceleratorState().deepspeed_plugin.deepspeed_config_process(must_match=True, **kwargs)
|
| 89 |
+
eval_logger.info("Detected that you are using DistributedType.DEEPSPEED. Make sure you run `accelerate config` and set zero stage to 0")
|
| 90 |
+
if accelerator.distributed_type == DistributedType.FSDP or accelerator.distributed_type == DistributedType.DEEPSPEED:
|
| 91 |
+
self._model = accelerator.prepare(self.model)
|
| 92 |
+
else:
|
| 93 |
+
self._model = accelerator.prepare_model(self.model, evaluation_mode=True)
|
| 94 |
+
self.accelerator = accelerator
|
| 95 |
+
if self.accelerator.is_local_main_process:
|
| 96 |
+
eval_logger.info(f"Using {accelerator.num_processes} devices with data parallelism")
|
| 97 |
+
self._rank = self.accelerator.local_process_index
|
| 98 |
+
self._world_size = self.accelerator.num_processes
|
| 99 |
+
elif accelerator.num_processes == 1 and device_map == "auto":
|
| 100 |
+
eval_logger.info(f"Using {accelerator.num_processes} devices with tensor parallelism")
|
| 101 |
+
self._rank = 0
|
| 102 |
+
self._world_size = 1
|
| 103 |
+
else:
|
| 104 |
+
eval_logger.info(f"Using single device: {self._device}")
|
| 105 |
+
self.model.to(self._device)
|
| 106 |
+
self._rank = 0
|
| 107 |
+
self._world_size = 1
|
| 108 |
+
|
| 109 |
+
@property
|
| 110 |
+
def config(self):
|
| 111 |
+
# return the associated transformers.AutoConfig for the given pretrained model.
|
| 112 |
+
return self._config
|
| 113 |
+
|
| 114 |
+
@property
|
| 115 |
+
def tokenizer(self):
|
| 116 |
+
return self._tokenizer
|
| 117 |
+
|
| 118 |
+
@property
|
| 119 |
+
def model(self):
|
| 120 |
+
# returns the model, unwrapping it if using Accelerate
|
| 121 |
+
if hasattr(self, "accelerator"):
|
| 122 |
+
return self.accelerator.unwrap_model(self._model)
|
| 123 |
+
else:
|
| 124 |
+
return self._model
|
| 125 |
+
|
| 126 |
+
@property
|
| 127 |
+
def eot_token_id(self):
|
| 128 |
+
# we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
|
| 129 |
+
return self.tokenizer.eos_token_id
|
| 130 |
+
|
| 131 |
+
@property
|
| 132 |
+
def max_length(self):
|
| 133 |
+
return self._max_length
|
| 134 |
+
|
| 135 |
+
def pad_sequence(self, input_ids, batch_first, padding_value):
|
| 136 |
+
if self.tokenizer.padding_side == "left":
|
| 137 |
+
input_ids = [torch.flip(_input_ids, [0]) for _input_ids in input_ids]
|
| 138 |
+
input_ids = torch.nn.utils.rnn.pad_sequence(input_ids, batch_first=batch_first, padding_value=padding_value)
|
| 139 |
+
if self.tokenizer.padding_side == "left":
|
| 140 |
+
input_ids = torch.flip(input_ids, [1])
|
| 141 |
+
return input_ids
|
| 142 |
+
|
| 143 |
+
@property
|
| 144 |
+
def batch_size(self):
|
| 145 |
+
return self.batch_size_per_gpu
|
| 146 |
+
|
| 147 |
+
@property
|
| 148 |
+
def device(self):
|
| 149 |
+
return self._device
|
| 150 |
+
|
| 151 |
+
@property
|
| 152 |
+
def rank(self):
|
| 153 |
+
return self._rank
|
| 154 |
+
|
| 155 |
+
@property
|
| 156 |
+
def world_size(self):
|
| 157 |
+
return self._world_size
|
| 158 |
+
|
| 159 |
+
def tok_encode(self, string: str, left_truncate_len=None, add_special_tokens=None) -> List[int]:
|
| 160 |
+
""" """
|
| 161 |
+
add_special_tokens = False if add_special_tokens is None else add_special_tokens
|
| 162 |
+
encoding = self.tokenizer.encode(string, add_special_tokens=add_special_tokens)
|
| 163 |
+
# left-truncate the encoded context to be at most `left_truncate_len` tokens long
|
| 164 |
+
if left_truncate_len:
|
| 165 |
+
encoding = encoding[-left_truncate_len:]
|
| 166 |
+
return encoding
|
| 167 |
+
|
| 168 |
+
def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
|
| 169 |
+
return super().loglikelihood(requests)
|
| 170 |
+
|
| 171 |
+
def flatten(self, input):
|
| 172 |
+
new_list = []
|
| 173 |
+
for i in input:
|
| 174 |
+
for j in i:
|
| 175 |
+
new_list.append(j)
|
| 176 |
+
return new_list
|
| 177 |
+
|
| 178 |
+
def generate_until(self, requests) -> List[str]:
|
| 179 |
+
res = []
|
| 180 |
+
pbar = tqdm(total=len(requests), disable=(self.rank != 0), desc="Model Responding")
|
| 181 |
+
|
| 182 |
+
for contexts, gen_kwargs, doc_to_visual, doc_id, task, split in [reg.args for reg in requests]:
|
| 183 |
+
# encode, pad, and truncate contexts for this batch
|
| 184 |
+
visuals = [doc_to_visual(self.task_dict[task][split][doc_id])]
|
| 185 |
+
visuals = self.flatten(visuals)
|
| 186 |
+
assert len(visuals) == 1
|
| 187 |
+
clip = read_video_pyav(visuals[0], self.num_frames)
|
| 188 |
+
|
| 189 |
+
inputs = self._processor(text=self.prompt.format(contexts), videos=clip, return_tensors="pt")
|
| 190 |
+
pixel_values_videos = inputs["pixel_values_videos"]
|
| 191 |
+
if pixel_values_videos.shape[1] != self.num_frames:
|
| 192 |
+
empty_frames = torch.zeros((1, self.num_frames - pixel_values_videos.shape[1], *pixel_values_videos.shape[2:]), dtype=pixel_values_videos.dtype)
|
| 193 |
+
pixel_values_videos = torch.cat([pixel_values_videos, empty_frames], dim=1)
|
| 194 |
+
inputs["pixel_values_videos"] = pixel_values_videos
|
| 195 |
+
inputs = {k: v.to(self.device) for k, v in inputs.items()}
|
| 196 |
+
|
| 197 |
+
if "max_new_tokens" not in gen_kwargs:
|
| 198 |
+
gen_kwargs["max_new_tokens"] = 1024
|
| 199 |
+
if "temperature" not in gen_kwargs:
|
| 200 |
+
gen_kwargs["temperature"] = 0
|
| 201 |
+
if "top_p" not in gen_kwargs:
|
| 202 |
+
gen_kwargs["top_p"] = None
|
| 203 |
+
if "num_beams" not in gen_kwargs:
|
| 204 |
+
gen_kwargs["num_beams"] = 1
|
| 205 |
+
|
| 206 |
+
generate_ids = self.model.generate(**inputs, max_new_tokens=gen_kwargs["max_new_tokens"], temperature=gen_kwargs["temperature"])
|
| 207 |
+
|
| 208 |
+
outputs = self._processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0].split("ASSISTANT:")[-1].strip()
|
| 209 |
+
res.append(outputs)
|
| 210 |
+
pbar.update(1)
|
| 211 |
+
return res
|
| 212 |
+
|
| 213 |
+
def generate_until_multi_round(self, requests) -> List[str]:
|
| 214 |
+
raise NotImplementedError("TODO: Implement multi-round generation")
|
LaViDa/eval/lmms_eval/tasks/activitynetqa/utils.py
ADDED
|
@@ -0,0 +1,287 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import ast
|
| 2 |
+
import datetime
|
| 3 |
+
import json
|
| 4 |
+
import os
|
| 5 |
+
import sys
|
| 6 |
+
import time
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
|
| 9 |
+
import numpy as np
|
| 10 |
+
import openai
|
| 11 |
+
import requests
|
| 12 |
+
import yaml
|
| 13 |
+
from decord import VideoReader, cpu
|
| 14 |
+
from loguru import logger as eval_logger
|
| 15 |
+
from openai import OpenAI
|
| 16 |
+
|
| 17 |
+
import lmms_eval.tasks._task_utils.file_utils as file_utils
|
| 18 |
+
|
| 19 |
+
with open(Path(__file__).parent / "_default_template_yaml", "r") as f:
|
| 20 |
+
raw_data = f.readlines()
|
| 21 |
+
safe_data = []
|
| 22 |
+
for i, line in enumerate(raw_data):
|
| 23 |
+
# remove function definition since yaml load cannot handle it
|
| 24 |
+
if "!function" not in line:
|
| 25 |
+
safe_data.append(line)
|
| 26 |
+
|
| 27 |
+
config = yaml.safe_load("".join(safe_data))
|
| 28 |
+
|
| 29 |
+
NUM_SECONDS_TO_SLEEP = 5
|
| 30 |
+
|
| 31 |
+
GPT_EVAL_MODEL_NAME = config["metadata"]["gpt_eval_model_name"]
|
| 32 |
+
|
| 33 |
+
API_TYPE = os.getenv("API_TYPE", "openai")
|
| 34 |
+
|
| 35 |
+
if API_TYPE == "openai":
|
| 36 |
+
API_URL = os.getenv("OPENAI_API_URL", "https://api.openai.com/v1/chat/completions")
|
| 37 |
+
API_KEY = os.getenv("OPENAI_API_KEY", "YOUR_API_KEY")
|
| 38 |
+
headers = {
|
| 39 |
+
"Authorization": f"Bearer {API_KEY}",
|
| 40 |
+
"Content-Type": "application/json",
|
| 41 |
+
}
|
| 42 |
+
|
| 43 |
+
# Unzip all the zip files to HF HOME cache dir
|
| 44 |
+
HF_HOME = os.environ["HF_HOME"]
|
| 45 |
+
cache_dir = config["dataset_kwargs"]["cache_dir"]
|
| 46 |
+
cache_dir = os.path.join(HF_HOME, cache_dir)
|
| 47 |
+
cache_dir = os.path.join(cache_dir, "all_test")
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
# Pass in video path here
|
| 51 |
+
# Can only work correctly with video dataset
|
| 52 |
+
def activitynetqa_doc_to_visual(doc):
|
| 53 |
+
video_path = os.path.join(cache_dir, f"v_{doc['video_name']}.mp4")
|
| 54 |
+
extensions = ["mp4", "webm", "mkv"]
|
| 55 |
+
for ext in extensions:
|
| 56 |
+
modified_path = video_path.replace("mp4", ext)
|
| 57 |
+
if os.path.exists(modified_path):
|
| 58 |
+
return [modified_path]
|
| 59 |
+
sys.exit(f"video path:{video_path} does not exist, please check")
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
# This is the place where format the question
|
| 63 |
+
def activitynetqa_doc_to_text(doc, lmms_eval_specific_kwargs=None):
|
| 64 |
+
if lmms_eval_specific_kwargs is None:
|
| 65 |
+
lmms_eval_specific_kwargs = {}
|
| 66 |
+
pre_prompt = ""
|
| 67 |
+
post_prompt = ""
|
| 68 |
+
if "pre_prompt" in lmms_eval_specific_kwargs:
|
| 69 |
+
pre_prompt = lmms_eval_specific_kwargs["pre_prompt"]
|
| 70 |
+
if "post_prompt" in lmms_eval_specific_kwargs:
|
| 71 |
+
post_prompt = lmms_eval_specific_kwargs["post_prompt"]
|
| 72 |
+
|
| 73 |
+
raw_question = doc["question"]
|
| 74 |
+
question = raw_question.capitalize() + "?"
|
| 75 |
+
|
| 76 |
+
# type_specific_prompts = {
|
| 77 |
+
# '3': "Please answer with 'yes' or 'no'.",
|
| 78 |
+
# '4': "Please state the color as a single word.",
|
| 79 |
+
# '7': "Please give the numerical answer."
|
| 80 |
+
# }
|
| 81 |
+
|
| 82 |
+
# doc_type = str(doc['type'])
|
| 83 |
+
# type_specific_prompt = type_specific_prompts.get(doc_type, "")
|
| 84 |
+
|
| 85 |
+
# return f"{pre_prompt}{question} {type_specific_prompt}{post_prompt}"
|
| 86 |
+
return f"{pre_prompt}{question}{post_prompt}"
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def activitynetqa_doc_to_answer(doc):
|
| 90 |
+
return doc["answer"]
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def get_eval(question, answer, pred, max_tokens: int, retries: int = 5):
|
| 94 |
+
global headers
|
| 95 |
+
|
| 96 |
+
messages = [
|
| 97 |
+
{
|
| 98 |
+
"role": "system",
|
| 99 |
+
"content": "You are an intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs. "
|
| 100 |
+
"Your task is to compare the predicted answer with the correct answer and determine if they match meaningfully. Here's how you can accomplish the task:"
|
| 101 |
+
"------"
|
| 102 |
+
"##INSTRUCTIONS: "
|
| 103 |
+
"- Focus on the meaningful match between the predicted answer and the correct answer.\n"
|
| 104 |
+
"- Consider synonyms or paraphrases as valid matches.\n"
|
| 105 |
+
"- Evaluate the correctness of the prediction compared to the answer.",
|
| 106 |
+
},
|
| 107 |
+
{
|
| 108 |
+
"role": "user",
|
| 109 |
+
"content": f"Please evaluate the following video-based question-answer pair:\n\n"
|
| 110 |
+
f"Question: {question}\n"
|
| 111 |
+
f"Correct Answer: {answer}\n"
|
| 112 |
+
f"Predicted Answer: {pred}\n\n"
|
| 113 |
+
"Provide your evaluation only as a yes/no and score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. "
|
| 114 |
+
"Please generate the response in the form of a Python dictionary string with keys 'pred' and 'score', where value of 'pred' is a string of 'yes' or 'no' and value of 'score' is in INTEGER, not STRING."
|
| 115 |
+
"DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. "
|
| 116 |
+
"For example, your response should look like this: {'pred': 'yes', 'score': 4.8}.",
|
| 117 |
+
},
|
| 118 |
+
]
|
| 119 |
+
|
| 120 |
+
payload = {
|
| 121 |
+
"model": GPT_EVAL_MODEL_NAME,
|
| 122 |
+
"messages": messages,
|
| 123 |
+
"temperature": 0,
|
| 124 |
+
"max_tokens": max_tokens,
|
| 125 |
+
}
|
| 126 |
+
|
| 127 |
+
for attempt in range(retries):
|
| 128 |
+
try:
|
| 129 |
+
response = requests.post(API_URL, headers=headers, json=payload, timeout=60)
|
| 130 |
+
response.raise_for_status() # Raises HTTPError for bad responses
|
| 131 |
+
try:
|
| 132 |
+
response_data = response.json() # Attempt to parse JSON
|
| 133 |
+
except requests.exceptions.JSONDecodeError:
|
| 134 |
+
eval_logger.error(f"JSON decode error on attempt {attempt + 1}. Response text: {response.text}")
|
| 135 |
+
continue # Skip to next retry
|
| 136 |
+
content = response_data["choices"][0]["message"]["content"].strip()
|
| 137 |
+
if content != "":
|
| 138 |
+
return content, response_data["model"]
|
| 139 |
+
# Handle HTTP errors separately
|
| 140 |
+
except requests.exceptions.HTTPError as e:
|
| 141 |
+
eval_logger.error(f"HTTP error on attempt {attempt + 1}: {e}")
|
| 142 |
+
# Handle other requests-related errors
|
| 143 |
+
except requests.exceptions.RequestException as e:
|
| 144 |
+
eval_logger.error(f"Request exception on attempt {attempt + 1}: {e}")
|
| 145 |
+
except Exception as e:
|
| 146 |
+
eval_logger.error(f"Unexpected error on attempt {attempt + 1}: {e}")
|
| 147 |
+
|
| 148 |
+
# Handle other unexpected errors
|
| 149 |
+
if attempt < retries - 1:
|
| 150 |
+
time.sleep(NUM_SECONDS_TO_SLEEP)
|
| 151 |
+
else: # If this was the last attempt, log and return empty
|
| 152 |
+
eval_logger.error(f"All {retries} attempts failed. Last error message: {e}")
|
| 153 |
+
return "", ""
|
| 154 |
+
|
| 155 |
+
return "", ""
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
def parse_score(review):
|
| 159 |
+
try:
|
| 160 |
+
# Convert the string representation of a dictionary to an actual dictionary
|
| 161 |
+
review = "{" + review.split("{")[1].split("}")[0] + "}"
|
| 162 |
+
review_dict = ast.literal_eval(review)
|
| 163 |
+
# import pdb;pdb.set_trace()
|
| 164 |
+
score_match = review_dict["score"]
|
| 165 |
+
score = int(score_match)
|
| 166 |
+
pred = review_dict["pred"]
|
| 167 |
+
if "yes" in pred.lower():
|
| 168 |
+
pred = "yes"
|
| 169 |
+
elif "no" in pred.lower():
|
| 170 |
+
pred = "no"
|
| 171 |
+
# pred = review_dict.get("pred", "no")
|
| 172 |
+
# score = review_dict.get("score", 0)
|
| 173 |
+
return [pred, score]
|
| 174 |
+
except SyntaxError as e:
|
| 175 |
+
eval_logger.error(f"Syntax error parsing the review string: {e}. Review content: {review}")
|
| 176 |
+
except ValueError as e:
|
| 177 |
+
eval_logger.error(f"Value error parsing the review string: {e}. Review content: {review}")
|
| 178 |
+
except Exception as e:
|
| 179 |
+
eval_logger.error(f"Unexpected error parsing the review string: {e}. Review content: {review}")
|
| 180 |
+
|
| 181 |
+
|
| 182 |
+
def activitynetqa_process_results(doc, result):
|
| 183 |
+
"""
|
| 184 |
+
Args:
|
| 185 |
+
doc: a instance of the eval dataset
|
| 186 |
+
results: [pred]
|
| 187 |
+
Returns:
|
| 188 |
+
a dictionary
|
| 189 |
+
"""
|
| 190 |
+
try:
|
| 191 |
+
question = doc["question"]
|
| 192 |
+
answer = doc["answer"]
|
| 193 |
+
pred = result[0]
|
| 194 |
+
|
| 195 |
+
# Assume get_eval returns a review and the model name, and parse_score parses this review
|
| 196 |
+
review, model_name = get_eval(question, answer, pred, 64)
|
| 197 |
+
scores = parse_score(review)
|
| 198 |
+
except Exception as e:
|
| 199 |
+
eval_logger.error(f"Error for Question ID: {doc.get('question_id', 'Unknown')}: {e}")
|
| 200 |
+
review = "Failed to Get a Proper Review."
|
| 201 |
+
model_name = "Failed Request"
|
| 202 |
+
scores = ["no", 0]
|
| 203 |
+
|
| 204 |
+
return {
|
| 205 |
+
"gpt_eval_score": {"video_name": doc["video_name"], "question": doc["question"], "answer": doc["answer"], "pred": pred, "question_id": doc["question_id"], "type": doc["type"], "Correctness": scores[0], "score": scores[1]},
|
| 206 |
+
"gpt_eval_accuracy": {"video_name": doc["video_name"], "question": doc["question"], "answer": doc["answer"], "pred": pred, "question_id": doc["question_id"], "type": doc["type"], "Correctness": scores[0], "score": scores[1]},
|
| 207 |
+
}
|
| 208 |
+
|
| 209 |
+
|
| 210 |
+
def activitynetqa_gpt_eval(results, args):
|
| 211 |
+
"""
|
| 212 |
+
Process the result file containing predictions, score them using GPT,
|
| 213 |
+
and save the results with added scores and correctness fields to a new file.
|
| 214 |
+
|
| 215 |
+
Args:
|
| 216 |
+
result_file_path: path to the JSON file with results to be evaluated
|
| 217 |
+
eval_file_path: path to save the JSON file with evaluated results
|
| 218 |
+
"""
|
| 219 |
+
|
| 220 |
+
evaluated_results = []
|
| 221 |
+
|
| 222 |
+
# Process each result to generate scores
|
| 223 |
+
for data_dict in results:
|
| 224 |
+
try:
|
| 225 |
+
question = data_dict.get("Q", "")
|
| 226 |
+
answer = data_dict.get("A", "")
|
| 227 |
+
pred = data_dict.get("pred", "")
|
| 228 |
+
|
| 229 |
+
# Assume get_eval returns a review and the model name, and parse_score parses this review
|
| 230 |
+
review, model_name = get_eval(question, answer, pred, 64)
|
| 231 |
+
scores = parse_score(review)
|
| 232 |
+
except Exception as e:
|
| 233 |
+
eval_logger.error(f"Error for Question ID: {data_dict.get('question_id', 'Unknown')}: {e}")
|
| 234 |
+
review = "Failed to Get a Proper Review."
|
| 235 |
+
model_name = "Failed Request"
|
| 236 |
+
scores = ["no", 0]
|
| 237 |
+
|
| 238 |
+
# Update the dictionary with the new entries
|
| 239 |
+
updated_dict = {"video_name": data_dict["video_name"], "Correctness": scores[0], "score": scores[1], "Q": question, "A": answer, "pred": pred, "question_id": data_dict.get("question_id"), "type": data_dict.get("type")}
|
| 240 |
+
evaluated_results.append(updated_dict)
|
| 241 |
+
|
| 242 |
+
return evaluated_results
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
# Factory into different aggregate
|
| 246 |
+
def activitynetqa_aggregate_score(results, args):
|
| 247 |
+
yes_count = 0
|
| 248 |
+
no_count = 0
|
| 249 |
+
total_score = 0
|
| 250 |
+
|
| 251 |
+
# Iterate over the results to count correctness and sum scores
|
| 252 |
+
for result_dict in results:
|
| 253 |
+
if "yes" in result_dict["Correctness"].lower():
|
| 254 |
+
yes_count += 1
|
| 255 |
+
elif "no" in result_dict["Correctness"].lower():
|
| 256 |
+
no_count += 1
|
| 257 |
+
|
| 258 |
+
total_score += int(result_dict["score"])
|
| 259 |
+
|
| 260 |
+
# Calculate accuracy and average score
|
| 261 |
+
accuracy = yes_count / (yes_count + no_count) if (yes_count + no_count) > 0 else 0
|
| 262 |
+
average_score = total_score / len(results) if results else 0
|
| 263 |
+
eval_logger.info(f"Accuracy: {accuracy}")
|
| 264 |
+
eval_logger.info(f"Average Score: {average_score}")
|
| 265 |
+
return average_score
|
| 266 |
+
|
| 267 |
+
|
| 268 |
+
def activitynetqa_aggregate_accuracy(results, args):
|
| 269 |
+
yes_count = 0
|
| 270 |
+
no_count = 0
|
| 271 |
+
total_score = 0
|
| 272 |
+
|
| 273 |
+
# Iterate over the results to count correctness and sum scores
|
| 274 |
+
for result_dict in results:
|
| 275 |
+
if "yes" in result_dict["Correctness"].lower():
|
| 276 |
+
yes_count += 1
|
| 277 |
+
elif "no" in result_dict["Correctness"].lower():
|
| 278 |
+
no_count += 1
|
| 279 |
+
|
| 280 |
+
total_score += int(result_dict["score"])
|
| 281 |
+
|
| 282 |
+
# Calculate accuracy and average score
|
| 283 |
+
accuracy = yes_count / (yes_count + no_count) if (yes_count + no_count) > 0 else 0
|
| 284 |
+
average_score = total_score / len(results) if results else 0
|
| 285 |
+
eval_logger.info(f"Accuracy: {accuracy}")
|
| 286 |
+
eval_logger.info(f"Average Score: {average_score}")
|
| 287 |
+
return accuracy * 100
|
LaViDa/eval/lmms_eval/tasks/perceptiontest/test/perceptiontest_mcppl.yaml
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
dataset_name: "mc_question"
|
| 2 |
+
task: "perceptiontest_test_mcppl"
|
| 3 |
+
test_split: test
|
| 4 |
+
output_type: multiple_choice
|
| 5 |
+
doc_to_visual: !function utils.perceptiontest_doc_to_visual
|
| 6 |
+
doc_to_text: "question"
|
| 7 |
+
doc_to_target: !function utils.perceptiontest_doc_to_answer_mc
|
| 8 |
+
doc_to_choice: !function utils.perceptiontest_doc_to_choice
|
| 9 |
+
process_results: !function utils.perceptiontest_process_results_mc_ppl
|
| 10 |
+
metric_list:
|
| 11 |
+
- metric: submission
|
| 12 |
+
aggregation: !function utils.perceptiontest_aggregate_mc_ppl
|
| 13 |
+
higher_is_better: true
|
| 14 |
+
include: _default_template_yaml
|
LaViDa/eval/lmms_eval/tasks/video_detail_description/_default_template_yaml
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
dataset_path: lmms-lab/VideoDetailDescription
|
| 2 |
+
dataset_kwargs:
|
| 3 |
+
token: True
|
| 4 |
+
video: True
|
| 5 |
+
cache_dir: videochatgpt
|
| 6 |
+
lmms_eval_specific_kwargs:
|
| 7 |
+
default:
|
| 8 |
+
pre_prompt: ""
|
| 9 |
+
post_prompt: ""
|
| 10 |
+
|
| 11 |
+
metadata:
|
| 12 |
+
version: 0.0
|
| 13 |
+
gpt_eval_model_name: gpt-3.5-turbo-0613
|
LaViDa/eval/lmms_eval/tasks/video_detail_description/utils.py
ADDED
|
@@ -0,0 +1,221 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import ast
|
| 2 |
+
import datetime
|
| 3 |
+
import json
|
| 4 |
+
import os
|
| 5 |
+
import sys
|
| 6 |
+
import time
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
|
| 9 |
+
import requests
|
| 10 |
+
import yaml
|
| 11 |
+
|
| 12 |
+
import lmms_eval.tasks._task_utils.file_utils as file_utils
|
| 13 |
+
|
| 14 |
+
with open(Path(__file__).parent / "_default_template_yaml", "r") as f:
|
| 15 |
+
raw_data = f.readlines()
|
| 16 |
+
safe_data = []
|
| 17 |
+
for i, line in enumerate(raw_data):
|
| 18 |
+
# remove function definition since yaml load cannot handle it
|
| 19 |
+
if "!function" not in line:
|
| 20 |
+
safe_data.append(line)
|
| 21 |
+
|
| 22 |
+
config = yaml.safe_load("".join(safe_data))
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
NUM_SECONDS_TO_SLEEP = 5
|
| 26 |
+
|
| 27 |
+
GPT_EVAL_MODEL_NAME = config["metadata"]["gpt_eval_model_name"]
|
| 28 |
+
|
| 29 |
+
API_TYPE = os.getenv("API_TYPE", "openai")
|
| 30 |
+
|
| 31 |
+
if API_TYPE == "openai":
|
| 32 |
+
API_URL = os.getenv("OPENAI_API_URL", "https://api.openai.com/v1/chat/completions")
|
| 33 |
+
API_KEY = os.getenv("OPENAI_API_KEY", "YOUR_API_KEY")
|
| 34 |
+
headers = {
|
| 35 |
+
"Authorization": f"Bearer {API_KEY}",
|
| 36 |
+
"Content-Type": "application/json",
|
| 37 |
+
}
|
| 38 |
+
|
| 39 |
+
# A bit ugly here
|
| 40 |
+
# But the idea is that we will unzip all the zip files
|
| 41 |
+
# To HF HOME cache dir
|
| 42 |
+
# And load it here
|
| 43 |
+
HF_HOME = os.environ["HF_HOME"]
|
| 44 |
+
cache_dir = config["dataset_kwargs"]["cache_dir"]
|
| 45 |
+
cache_dir = os.path.join(HF_HOME, cache_dir)
|
| 46 |
+
cache_dir = os.path.join(cache_dir, "Test_Videos")
|
| 47 |
+
|
| 48 |
+
from loguru import logger as eval_logger
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
# Pass in video path here
|
| 52 |
+
# Can only work correctly with video llm
|
| 53 |
+
def video_detail_description_doc_to_visual(doc):
|
| 54 |
+
video_path = doc["video_name"] + ".mp4"
|
| 55 |
+
video_path = os.path.join(cache_dir, video_path)
|
| 56 |
+
if os.path.exists(video_path):
|
| 57 |
+
video_path = video_path
|
| 58 |
+
elif os.path.exists(video_path.replace("mp4", "MP4")):
|
| 59 |
+
video_path = video_path.replace("mp4", "MP4")
|
| 60 |
+
elif os.path.exists(video_path.replace("mp4", "mkv")):
|
| 61 |
+
video_path = video_path.replace("mp4", "mkv")
|
| 62 |
+
else:
|
| 63 |
+
sys.exit(f"video path:{video_path} does not exist, please check")
|
| 64 |
+
return [video_path]
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
# format the question
|
| 68 |
+
def video_detail_description_doc_to_text(doc, lmms_eval_specific_kwargs=None):
|
| 69 |
+
if lmms_eval_specific_kwargs is None:
|
| 70 |
+
lmms_eval_specific_kwargs = {}
|
| 71 |
+
pre_prompt = ""
|
| 72 |
+
post_prompt = ""
|
| 73 |
+
if "pre_prompt" in lmms_eval_specific_kwargs:
|
| 74 |
+
pre_prompt = lmms_eval_specific_kwargs["pre_prompt"]
|
| 75 |
+
if "post_prompt" in lmms_eval_specific_kwargs:
|
| 76 |
+
post_prompt = lmms_eval_specific_kwargs["post_prompt"]
|
| 77 |
+
|
| 78 |
+
question = doc["question"]
|
| 79 |
+
return f"{pre_prompt}{question}{post_prompt}"
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def video_detail_description_doc_to_answer(doc):
|
| 83 |
+
return doc["answer"]
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def get_eval_generic(question, answer, pred, max_tokens: int, retries: int = 5):
|
| 87 |
+
global headers
|
| 88 |
+
|
| 89 |
+
messages = [
|
| 90 |
+
{
|
| 91 |
+
"role": "system",
|
| 92 |
+
"content": "You are an intelligent chatbot designed for evaluating the detail orientation of generative outputs for video-based question-answer pairs. "
|
| 93 |
+
"Your task is to compare the predicted answer with the correct answer and determine its level of detail, considering both completeness and specificity. Here's how you can accomplish the task:"
|
| 94 |
+
"------"
|
| 95 |
+
"##INSTRUCTIONS: "
|
| 96 |
+
"- Check if the predicted answer covers all major points from the video. The response should not leave out any key aspects.\n"
|
| 97 |
+
"- Evaluate whether the predicted answer includes specific details rather than just generic points. It should provide comprehensive information that is tied to specific elements of the video.\n"
|
| 98 |
+
"- Consider synonyms or paraphrases as valid matches.\n"
|
| 99 |
+
"- Provide a single evaluation score that reflects the level of detail orientation of the prediction, considering both completeness and specificity.",
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"role": "user",
|
| 103 |
+
"content": "Please evaluate the following video-based question-answer pair:\n\n"
|
| 104 |
+
f"Question: {question}\n"
|
| 105 |
+
f"Correct Answer: {answer}\n"
|
| 106 |
+
f"Predicted Answer: {pred}\n\n"
|
| 107 |
+
"Provide your evaluation only as a detail orientation score where the detail orientation score is an integer value between 0 and 5, with 5 indicating the highest level of detail orientation. "
|
| 108 |
+
"Please generate the response in the form of a Python dictionary string with keys 'score', where its value is the detail orientation score in INTEGER, not STRING."
|
| 109 |
+
"DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. "
|
| 110 |
+
"For example, your response should look like this: {'score': 4.8}.",
|
| 111 |
+
},
|
| 112 |
+
]
|
| 113 |
+
|
| 114 |
+
payload = {
|
| 115 |
+
"model": GPT_EVAL_MODEL_NAME,
|
| 116 |
+
"messages": messages,
|
| 117 |
+
"temperature": 0,
|
| 118 |
+
"max_tokens": max_tokens,
|
| 119 |
+
# "response_format": {"type": "json_object"},
|
| 120 |
+
}
|
| 121 |
+
|
| 122 |
+
for attempt in range(retries):
|
| 123 |
+
try:
|
| 124 |
+
response = requests.post(API_URL, headers=headers, json=payload, timeout=60)
|
| 125 |
+
response.raise_for_status() # Raises HTTPError for bad responses
|
| 126 |
+
try:
|
| 127 |
+
response_data = response.json() # Attempt to parse JSON
|
| 128 |
+
except requests.exceptions.JSONDecodeError:
|
| 129 |
+
eval_logger.error(f"JSON decode error on attempt {attempt + 1}. Response text: {response.text}")
|
| 130 |
+
continue # Skip to next retry
|
| 131 |
+
content = response_data["choices"][0]["message"]["content"].strip()
|
| 132 |
+
if content != "":
|
| 133 |
+
return content, response_data["model"]
|
| 134 |
+
# Handle HTTP errors separately
|
| 135 |
+
except requests.exceptions.HTTPError as e:
|
| 136 |
+
eval_logger.error(f"HTTP error on attempt {attempt + 1}: {e}")
|
| 137 |
+
# Handle other requests-related errors
|
| 138 |
+
except requests.exceptions.RequestException as e:
|
| 139 |
+
eval_logger.error(f"Request exception on attempt {attempt + 1}: {e}")
|
| 140 |
+
except Exception as e:
|
| 141 |
+
eval_logger.error(f"Unexpected error on attempt {attempt + 1}: {e}")
|
| 142 |
+
|
| 143 |
+
if "Sorry! We've encountered an issue with repetitive patterns in your prompt. Please try again with a different prompt." in json.loads(response.content)["error"]["message"]:
|
| 144 |
+
eval_logger.error(f"Repetitive patterns in prompt. Drop this data.")
|
| 145 |
+
return "", ""
|
| 146 |
+
|
| 147 |
+
# Handle other unexpected errors
|
| 148 |
+
if attempt < retries - 1:
|
| 149 |
+
time.sleep(NUM_SECONDS_TO_SLEEP)
|
| 150 |
+
else: # If this was the last attempt, log and return empty
|
| 151 |
+
eval_logger.error(f"All {retries} attempts failed.")
|
| 152 |
+
return "", ""
|
| 153 |
+
|
| 154 |
+
return "", ""
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
def parse_score(review):
|
| 158 |
+
try:
|
| 159 |
+
# Convert the string representation of a dictionary to an actual dictionary
|
| 160 |
+
review_dict = ast.literal_eval(review)
|
| 161 |
+
score = review_dict.get("score", 0)
|
| 162 |
+
return int(score)
|
| 163 |
+
except SyntaxError as e:
|
| 164 |
+
eval_logger.error(f"Syntax error parsing the review string: {e}. Review content: {review}")
|
| 165 |
+
return 0
|
| 166 |
+
except ValueError as e:
|
| 167 |
+
eval_logger.error(f"Value error parsing the review string: {e}. Review content: {review}")
|
| 168 |
+
return 0
|
| 169 |
+
except Exception as e:
|
| 170 |
+
eval_logger.error(f"Unexpected error parsing the review string: {e}. Review content: {review}")
|
| 171 |
+
return 0
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
def gpt_eval(data_dict):
|
| 175 |
+
evaluated_results = []
|
| 176 |
+
|
| 177 |
+
try:
|
| 178 |
+
question = data_dict["question"]
|
| 179 |
+
answer = data_dict["answer"]
|
| 180 |
+
pred = data_dict["pred"]
|
| 181 |
+
|
| 182 |
+
# Assume get_eval returns a review and the model name, and parse_score parses this review
|
| 183 |
+
review, model_name = get_eval_generic(question, answer, pred, 64)
|
| 184 |
+
score = parse_score(review)
|
| 185 |
+
except Exception as e:
|
| 186 |
+
eval_logger.error(f"Error for Video Name: {data_dict.get('video_name', 'Unknown')}: {e}")
|
| 187 |
+
review = "Failed to Get a Proper Review."
|
| 188 |
+
model_name = ""
|
| 189 |
+
score = 0
|
| 190 |
+
|
| 191 |
+
# Update the dictionary with the new entries
|
| 192 |
+
updated_dict = {
|
| 193 |
+
"video_name": data_dict["video_name"],
|
| 194 |
+
"review": review,
|
| 195 |
+
"score": score,
|
| 196 |
+
}
|
| 197 |
+
|
| 198 |
+
return updated_dict
|
| 199 |
+
|
| 200 |
+
|
| 201 |
+
# Process result for evaluation in generic task
|
| 202 |
+
def video_detail_description_process_results_generic(doc, result):
|
| 203 |
+
pred = result[0]
|
| 204 |
+
doc["pred"] = pred
|
| 205 |
+
eval_results = gpt_eval(doc)
|
| 206 |
+
|
| 207 |
+
return {"gpt_eval_score": {"video_name": doc["video_name"], "question": doc["question"], "answer": doc["answer"], "pred": pred, "score": eval_results["score"], "review": eval_results["review"]}}
|
| 208 |
+
|
| 209 |
+
|
| 210 |
+
def video_detail_description_aggregate_score(results, args):
|
| 211 |
+
score = 0
|
| 212 |
+
for result in results:
|
| 213 |
+
eval_score = result["score"]
|
| 214 |
+
try:
|
| 215 |
+
eval_score = int(eval_score)
|
| 216 |
+
except:
|
| 217 |
+
eval_score = 0.0
|
| 218 |
+
|
| 219 |
+
score += eval_score
|
| 220 |
+
|
| 221 |
+
return score / len(results)
|
LaViDa/eval/lmms_eval/tasks/video_detail_description/video_detail_description.yaml
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
task: "video_dc499"
|
| 2 |
+
test_split: test
|
| 3 |
+
output_type: generate_until
|
| 4 |
+
doc_to_visual: !function utils.video_detail_description_doc_to_visual
|
| 5 |
+
doc_to_text: !function utils.video_detail_description_doc_to_text
|
| 6 |
+
doc_to_target: !function utils.video_detail_description_doc_to_answer
|
| 7 |
+
process_results: !function utils.video_detail_description_process_results_generic
|
| 8 |
+
metric_list:
|
| 9 |
+
- metric: gpt_eval_score
|
| 10 |
+
aggregation: !function utils.video_detail_description_aggregate_score
|
| 11 |
+
higher_is_better: true
|
| 12 |
+
include: _default_template_yaml
|
LaViDa/eval/lmms_eval/utils.py
ADDED
|
@@ -0,0 +1,1018 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import collections
|
| 2 |
+
import datetime
|
| 3 |
+
import fnmatch
|
| 4 |
+
import functools
|
| 5 |
+
import hashlib
|
| 6 |
+
import importlib.util
|
| 7 |
+
import inspect
|
| 8 |
+
import json
|
| 9 |
+
import os
|
| 10 |
+
import pathlib
|
| 11 |
+
import re
|
| 12 |
+
import subprocess
|
| 13 |
+
import sys
|
| 14 |
+
import warnings
|
| 15 |
+
from typing import (
|
| 16 |
+
Any,
|
| 17 |
+
Callable,
|
| 18 |
+
Iterable,
|
| 19 |
+
Iterator,
|
| 20 |
+
List,
|
| 21 |
+
Literal,
|
| 22 |
+
Optional,
|
| 23 |
+
Tuple,
|
| 24 |
+
Type,
|
| 25 |
+
Union,
|
| 26 |
+
)
|
| 27 |
+
|
| 28 |
+
import yaml
|
| 29 |
+
|
| 30 |
+
warnings.simplefilter("ignore", category=DeprecationWarning)
|
| 31 |
+
warnings.filterwarnings("ignore")
|
| 32 |
+
|
| 33 |
+
import gc
|
| 34 |
+
from itertools import islice
|
| 35 |
+
|
| 36 |
+
import numpy as np
|
| 37 |
+
import pytz
|
| 38 |
+
import torch
|
| 39 |
+
import transformers
|
| 40 |
+
from jinja2 import BaseLoader, Environment, StrictUndefined
|
| 41 |
+
from loguru import logger as eval_logger
|
| 42 |
+
|
| 43 |
+
SPACING = " " * 47
|
| 44 |
+
HIGHER_IS_BETTER_SYMBOLS = {
|
| 45 |
+
True: "↑",
|
| 46 |
+
False: "↓",
|
| 47 |
+
}
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def is_json(string):
|
| 51 |
+
try:
|
| 52 |
+
json.loads(string)
|
| 53 |
+
return True
|
| 54 |
+
except json.JSONDecodeError:
|
| 55 |
+
return False
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def hash_string(string: str) -> str:
|
| 59 |
+
return hashlib.sha256(string.encode("utf-8")).hexdigest()
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def escaped_split(text, sep_char, maxsplit=-1):
|
| 63 |
+
"""Split text into a list on occurrences of the given separation
|
| 64 |
+
character `sep_char`. The separation character may be escaped by a
|
| 65 |
+
backslash to avoid splitting at that location.
|
| 66 |
+
|
| 67 |
+
The separation character must be a string of size 1.
|
| 68 |
+
|
| 69 |
+
If `maxsplit` is given, at most `maxsplit` splits are done (thus,
|
| 70 |
+
the list will have at most `maxsplit + 1` elements). If `maxsplit`
|
| 71 |
+
is not specified or less than 0, then there is no limit on the
|
| 72 |
+
number of splits (all possible splits are made).
|
| 73 |
+
"""
|
| 74 |
+
assert len(sep_char) == 1, "separation string must be a single character for escaped splitting"
|
| 75 |
+
|
| 76 |
+
if maxsplit == 0:
|
| 77 |
+
return text
|
| 78 |
+
maxsplit = max(0, maxsplit)
|
| 79 |
+
|
| 80 |
+
return re.split(r"(?<!\\)" + sep_char, text, maxsplit)
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def handle_arg_string(arg):
|
| 84 |
+
if arg.lower() == "true":
|
| 85 |
+
return True
|
| 86 |
+
elif arg.lower() == "false":
|
| 87 |
+
return False
|
| 88 |
+
elif arg.isnumeric():
|
| 89 |
+
return int(arg)
|
| 90 |
+
try:
|
| 91 |
+
return float(arg)
|
| 92 |
+
except ValueError:
|
| 93 |
+
return arg
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
def handle_non_serializable(o):
|
| 97 |
+
if isinstance(o, np.int64) or isinstance(o, np.int32):
|
| 98 |
+
return int(o)
|
| 99 |
+
elif isinstance(o, set):
|
| 100 |
+
return list(o)
|
| 101 |
+
else:
|
| 102 |
+
return str(o)
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def sanitize_list(sub):
|
| 106 |
+
"""
|
| 107 |
+
Takes possible nested list and recursively converts all inner component to strings
|
| 108 |
+
"""
|
| 109 |
+
if isinstance(sub, list):
|
| 110 |
+
return [sanitize_list(item) for item in sub]
|
| 111 |
+
if isinstance(sub, tuple):
|
| 112 |
+
return tuple(sanitize_list(item) for item in sub)
|
| 113 |
+
else:
|
| 114 |
+
return str(sub)
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def simple_parse_args_string(args_string):
|
| 118 |
+
"""
|
| 119 |
+
Parses something like
|
| 120 |
+
args1=val1,arg2=val2
|
| 121 |
+
Into a dictionary
|
| 122 |
+
"""
|
| 123 |
+
args_string = args_string.strip()
|
| 124 |
+
if not args_string:
|
| 125 |
+
return {}
|
| 126 |
+
arg_list = [arg for arg in args_string.split(",") if arg]
|
| 127 |
+
args_dict = {k: handle_arg_string(v) for k, v in [arg.split("=") for arg in arg_list]}
|
| 128 |
+
return args_dict
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def join_iters(iters):
|
| 132 |
+
for iter in iters:
|
| 133 |
+
yield from iter
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def chunks(iter, n: int = 0, fn=None):
|
| 137 |
+
"""
|
| 138 |
+
Divides an iterable into chunks of specified size or based on a given function.
|
| 139 |
+
Useful for batching
|
| 140 |
+
|
| 141 |
+
Parameters:
|
| 142 |
+
- iter: The input iterable to be divided into chunks.
|
| 143 |
+
- n: An integer representing the size of each chunk. Default is 0.
|
| 144 |
+
- fn: A function that takes the current index and the iterable as arguments and returns the size of the chunk. Default is None.
|
| 145 |
+
|
| 146 |
+
Returns:
|
| 147 |
+
An iterator that yields chunks of the input iterable.
|
| 148 |
+
|
| 149 |
+
Example usage:
|
| 150 |
+
```
|
| 151 |
+
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
|
| 152 |
+
for chunk in chunks(data, 3):
|
| 153 |
+
print(chunk)
|
| 154 |
+
```
|
| 155 |
+
Output:
|
| 156 |
+
```
|
| 157 |
+
[1, 2, 3]
|
| 158 |
+
[4, 5, 6]
|
| 159 |
+
[7, 8, 9]
|
| 160 |
+
[10]
|
| 161 |
+
```
|
| 162 |
+
"""
|
| 163 |
+
arr = []
|
| 164 |
+
for i, x in enumerate(iter):
|
| 165 |
+
arr.append(x)
|
| 166 |
+
if len(arr) == (fn(i, iter) if fn else n):
|
| 167 |
+
yield arr
|
| 168 |
+
arr = []
|
| 169 |
+
|
| 170 |
+
if arr:
|
| 171 |
+
yield arr
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
def group(arr, fn):
|
| 175 |
+
res = collections.defaultdict(list)
|
| 176 |
+
|
| 177 |
+
for ob in arr:
|
| 178 |
+
res[fn(ob)].append(ob)
|
| 179 |
+
|
| 180 |
+
return list(res.values())
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
class MultiChoice:
|
| 184 |
+
def __init__(self, choices) -> None:
|
| 185 |
+
self.choices = choices
|
| 186 |
+
|
| 187 |
+
# Simple wildcard support (linux filename patterns)
|
| 188 |
+
def __contains__(self, values) -> bool:
|
| 189 |
+
for value in values.split(","):
|
| 190 |
+
if len(fnmatch.filter(self.choices, value)) == 0:
|
| 191 |
+
eval_logger.info(f"Available tasks to choose:")
|
| 192 |
+
for choice in self.choices:
|
| 193 |
+
eval_logger.info(f" - {choice}")
|
| 194 |
+
raise ValueError("'{}' is not in task list".format(value))
|
| 195 |
+
return True
|
| 196 |
+
|
| 197 |
+
def __iter__(self) -> Iterator:
|
| 198 |
+
for choice in self.choices:
|
| 199 |
+
yield choice
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
# Returns a list containing all values of the source_list that
|
| 203 |
+
# match at least one of the patterns
|
| 204 |
+
def pattern_match(patterns, source_list):
|
| 205 |
+
if type(patterns) == str:
|
| 206 |
+
patterns = [patterns]
|
| 207 |
+
|
| 208 |
+
task_names = set()
|
| 209 |
+
for pattern in patterns:
|
| 210 |
+
try:
|
| 211 |
+
for matching in fnmatch.filter(source_list, pattern):
|
| 212 |
+
task_names.add(matching)
|
| 213 |
+
except Exception as e:
|
| 214 |
+
eval_logger.error(f"Error matching pattern {pattern}: {e}")
|
| 215 |
+
return sorted(list(task_names))
|
| 216 |
+
|
| 217 |
+
|
| 218 |
+
def general_detokenize(string):
|
| 219 |
+
string = string.replace(" n't", "n't")
|
| 220 |
+
string = string.replace(" )", ")")
|
| 221 |
+
string = string.replace("( ", "(")
|
| 222 |
+
string = string.replace('" ', '"')
|
| 223 |
+
string = string.replace(' "', '"')
|
| 224 |
+
string = re.sub(r" (['.,])", r"\1", string)
|
| 225 |
+
return string
|
| 226 |
+
|
| 227 |
+
|
| 228 |
+
def get_file_task_name(filename: str) -> str:
|
| 229 |
+
"""
|
| 230 |
+
Given the sample results filenames, extracts and returns the task name.
|
| 231 |
+
"""
|
| 232 |
+
return filename[filename.find("_") + 1 : filename.rfind("_")]
|
| 233 |
+
|
| 234 |
+
|
| 235 |
+
def get_file_datetime(filename: str) -> str:
|
| 236 |
+
"""
|
| 237 |
+
Given the results and sample results filenames, extracts and returns the datetime.
|
| 238 |
+
"""
|
| 239 |
+
return filename[filename.rfind("_") + 1 :].replace(".jsonl", "")
|
| 240 |
+
|
| 241 |
+
|
| 242 |
+
def sanitize_model_name(model_name: str, full_path: bool = False) -> str:
|
| 243 |
+
"""
|
| 244 |
+
Given the model name, returns a sanitized version of it.
|
| 245 |
+
"""
|
| 246 |
+
if full_path:
|
| 247 |
+
return re.sub(r"[\"<>:/\|\\?\*\[\]]+", "__", model_name)
|
| 248 |
+
else:
|
| 249 |
+
parts = model_name.split("/")
|
| 250 |
+
last_two = "/".join(parts[-2:]) if len(parts) > 1 else parts[-1] # accommondate for models that are in Hugging Face Hub format like lmms-lab/llava-onevision-qwen2-0.5b
|
| 251 |
+
return re.sub(r"[\"<>:/\|\\?\*\[\]]+", "__", last_two)
|
| 252 |
+
|
| 253 |
+
|
| 254 |
+
def sanitize_task_name(task_name: str) -> str:
|
| 255 |
+
"""
|
| 256 |
+
Given the task name, returns a sanitized version of it.
|
| 257 |
+
"""
|
| 258 |
+
return re.sub(r"\W", "_", task_name)
|
| 259 |
+
|
| 260 |
+
|
| 261 |
+
def get_latest_filename(filenames: List[str]) -> str:
|
| 262 |
+
"""
|
| 263 |
+
Given a list of filenames, returns the filename with the latest datetime.
|
| 264 |
+
"""
|
| 265 |
+
return max(filenames, key=lambda f: get_file_datetime(f))
|
| 266 |
+
|
| 267 |
+
|
| 268 |
+
def get_results_filenames(filenames: List[str]) -> List[str]:
|
| 269 |
+
"""
|
| 270 |
+
Extracts filenames that correspond to aggregated results.
|
| 271 |
+
"""
|
| 272 |
+
return [f for f in filenames if "results" in f and ".json" in f]
|
| 273 |
+
|
| 274 |
+
|
| 275 |
+
def get_sample_results_filenames(filenames: List[str]) -> List[str]:
|
| 276 |
+
"""
|
| 277 |
+
Extracts filenames that correspond to sample results.
|
| 278 |
+
"""
|
| 279 |
+
return [f for f in filenames if "/samples_" in f and ".json" in f]
|
| 280 |
+
|
| 281 |
+
|
| 282 |
+
def get_rolling_token_windows(token_list, prefix_token, max_seq_len, context_len):
|
| 283 |
+
"""
|
| 284 |
+
- context_len allows for a rolling window context, allowing each prediction window to potentially
|
| 285 |
+
condition on some context
|
| 286 |
+
|
| 287 |
+
:param token_list: list
|
| 288 |
+
List of tokens to be PREDICTED
|
| 289 |
+
:param max_seq_len: int
|
| 290 |
+
max_seq_len of model (or max_seq_len we want to use)
|
| 291 |
+
:param context_len: int
|
| 292 |
+
Amount of desired token context for prediction. Needs to be at least 1.
|
| 293 |
+
:param prefix_token: token
|
| 294 |
+
Dummy token like <eos> so the first token has something to condition on
|
| 295 |
+
:return: generator
|
| 296 |
+
Generator of tuples
|
| 297 |
+
(input_tokens, pred_tokens)
|
| 298 |
+
Note: Score only the last len(pred_tokens) logits of the LMM
|
| 299 |
+
"""
|
| 300 |
+
assert 1 <= context_len <= max_seq_len
|
| 301 |
+
if not token_list:
|
| 302 |
+
return
|
| 303 |
+
# +1 offset, going from input->preds
|
| 304 |
+
pred_len = max_seq_len - context_len + 1
|
| 305 |
+
predicted = 0
|
| 306 |
+
|
| 307 |
+
# Special handling for first window: predict all tokens
|
| 308 |
+
first_seq_len = min(max_seq_len, len(token_list))
|
| 309 |
+
yield ([prefix_token] + token_list[: first_seq_len - 1], token_list[:first_seq_len])
|
| 310 |
+
predicted += first_seq_len
|
| 311 |
+
|
| 312 |
+
while predicted < len(token_list):
|
| 313 |
+
window_pred_len = min(len(token_list) - predicted, pred_len)
|
| 314 |
+
window_end = predicted + window_pred_len
|
| 315 |
+
|
| 316 |
+
yield (
|
| 317 |
+
token_list[window_end - max_seq_len - 1 : window_end - 1],
|
| 318 |
+
token_list[window_end - window_pred_len : window_end],
|
| 319 |
+
)
|
| 320 |
+
predicted += window_pred_len
|
| 321 |
+
|
| 322 |
+
|
| 323 |
+
def make_disjoint_window(pair):
|
| 324 |
+
"""Takes output from get_rolling_token_windows and makes the context not overlap with the continuation"""
|
| 325 |
+
a, b = pair
|
| 326 |
+
return a[: len(a) - (len(b) - 1)], b
|
| 327 |
+
|
| 328 |
+
|
| 329 |
+
class EnhancedJSONEncoder(json.JSONEncoder):
|
| 330 |
+
"""
|
| 331 |
+
Provides a proper json encoding for the loggers and trackers json dumps.
|
| 332 |
+
Notably manages the json encoding of dataclasses.
|
| 333 |
+
"""
|
| 334 |
+
|
| 335 |
+
def default(self, o):
|
| 336 |
+
if is_dataclass(o):
|
| 337 |
+
return asdict(o)
|
| 338 |
+
return super().default(o)
|
| 339 |
+
|
| 340 |
+
|
| 341 |
+
class Reorderer:
|
| 342 |
+
def __init__(self, arr: List[Any], fn: Callable) -> None:
|
| 343 |
+
"""Reorder an array according to some function
|
| 344 |
+
|
| 345 |
+
Args:
|
| 346 |
+
arr (List[Any]): The initial array
|
| 347 |
+
fn (Callable[[Any], Any]): A function to determine the priority of elements
|
| 348 |
+
"""
|
| 349 |
+
self.size = len(arr)
|
| 350 |
+
arr = list(enumerate(arr))
|
| 351 |
+
arr = group(arr, lambda x: fn(x[1]))
|
| 352 |
+
# arr = [([y[0] for y in x], x[0][1]) for x in arr]
|
| 353 |
+
# TODO: overhaul reorderer. It currently grouped requests by content but we don't want this
|
| 354 |
+
arr = [([y[0]], x[0][1]) for x in arr for y in x]
|
| 355 |
+
arr.sort(key=lambda x: fn(x[1]))
|
| 356 |
+
|
| 357 |
+
self.arr = arr
|
| 358 |
+
|
| 359 |
+
def get_reordered(self):
|
| 360 |
+
"""Gets the reordered array
|
| 361 |
+
|
| 362 |
+
Returns:
|
| 363 |
+
List[Any]: The reordered array
|
| 364 |
+
"""
|
| 365 |
+
return [x[1] for x in self.arr]
|
| 366 |
+
|
| 367 |
+
def get_original(self, newarr):
|
| 368 |
+
"""Restores the original order of a new array based on the old array's order
|
| 369 |
+
|
| 370 |
+
Args:
|
| 371 |
+
newarr (List[Any]): The array to be restored
|
| 372 |
+
|
| 373 |
+
Returns:
|
| 374 |
+
List[Any]: The array restored to the original order
|
| 375 |
+
"""
|
| 376 |
+
res = [None] * self.size
|
| 377 |
+
cov = [False] * self.size
|
| 378 |
+
|
| 379 |
+
for (inds, _), v in zip(self.arr, newarr):
|
| 380 |
+
for ind in inds:
|
| 381 |
+
res[ind] = v
|
| 382 |
+
cov[ind] = True
|
| 383 |
+
|
| 384 |
+
assert all(cov)
|
| 385 |
+
|
| 386 |
+
return res
|
| 387 |
+
|
| 388 |
+
|
| 389 |
+
class Grouper:
|
| 390 |
+
"""
|
| 391 |
+
takes an array `arr` and function `fn` and returns a dictionary
|
| 392 |
+
with keys fn(ob) for each ob in `arr` and with values `self.arr[key]` a list of all
|
| 393 |
+
objects in `arr` satisfying `key == fn(ob)`.
|
| 394 |
+
"""
|
| 395 |
+
|
| 396 |
+
def __init__(self, arr, fn) -> None:
|
| 397 |
+
# self.orig_arr = arr
|
| 398 |
+
self.size = len(arr)
|
| 399 |
+
arr = list(enumerate(arr))
|
| 400 |
+
|
| 401 |
+
def group_return_dict(arr, fn):
|
| 402 |
+
res = collections.defaultdict(list)
|
| 403 |
+
|
| 404 |
+
for ob in arr:
|
| 405 |
+
res[fn(ob)].append(ob)
|
| 406 |
+
return res
|
| 407 |
+
|
| 408 |
+
arr = group_return_dict(arr, lambda x: fn(x[1]))
|
| 409 |
+
|
| 410 |
+
# self.arr has format Dict[Tuple[int, <entry from orig. arr>]]
|
| 411 |
+
self.arr = arr
|
| 412 |
+
self._grouped = None
|
| 413 |
+
|
| 414 |
+
def get_grouped(self):
|
| 415 |
+
# return the contents but not indices for our grouped dict.
|
| 416 |
+
if self._grouped:
|
| 417 |
+
return self._grouped
|
| 418 |
+
grouped = {}
|
| 419 |
+
for key in self.arr.keys():
|
| 420 |
+
# drop the index from each element of self.arr
|
| 421 |
+
grouped[key] = [y[1] for y in self.arr[key]]
|
| 422 |
+
self._grouped = grouped
|
| 423 |
+
return grouped
|
| 424 |
+
|
| 425 |
+
def get_original(self, grouped_dict):
|
| 426 |
+
# take in a grouped dictionary with e.g. results for each key listed
|
| 427 |
+
# in the same order as the instances in `self.arr`, and
|
| 428 |
+
# return the results in the same (single list) order as `self.orig_arr`.
|
| 429 |
+
res = [None] * self.size
|
| 430 |
+
cov = [False] * self.size
|
| 431 |
+
# orig = [None] * self.size
|
| 432 |
+
|
| 433 |
+
assert grouped_dict.keys() == self.arr.keys()
|
| 434 |
+
|
| 435 |
+
for key in grouped_dict.keys():
|
| 436 |
+
for (ind, _), v in zip(self.arr[key], grouped_dict[key]):
|
| 437 |
+
res[ind] = v
|
| 438 |
+
cov[ind] = True
|
| 439 |
+
# orig[ind] = _
|
| 440 |
+
|
| 441 |
+
assert all(cov)
|
| 442 |
+
# assert orig == self.orig_arr
|
| 443 |
+
|
| 444 |
+
return res
|
| 445 |
+
|
| 446 |
+
|
| 447 |
+
def make_table(result_dict, column: str = "results", sort_results: bool = False):
|
| 448 |
+
"""Generate table of results."""
|
| 449 |
+
from pytablewriter import LatexTableWriter, MarkdownTableWriter
|
| 450 |
+
|
| 451 |
+
if column == "results":
|
| 452 |
+
column_name = "Tasks"
|
| 453 |
+
elif column == "groups":
|
| 454 |
+
column_name = "Groups"
|
| 455 |
+
|
| 456 |
+
all_headers = [
|
| 457 |
+
column_name,
|
| 458 |
+
"Version",
|
| 459 |
+
"Filter",
|
| 460 |
+
"n-shot",
|
| 461 |
+
"Metric",
|
| 462 |
+
"",
|
| 463 |
+
"Value",
|
| 464 |
+
"",
|
| 465 |
+
"Stderr",
|
| 466 |
+
]
|
| 467 |
+
|
| 468 |
+
md_writer = MarkdownTableWriter()
|
| 469 |
+
latex_writer = LatexTableWriter()
|
| 470 |
+
md_writer.headers = all_headers
|
| 471 |
+
latex_writer.headers = all_headers
|
| 472 |
+
|
| 473 |
+
values = []
|
| 474 |
+
|
| 475 |
+
keys = result_dict[column].keys()
|
| 476 |
+
if sort_results:
|
| 477 |
+
# sort entries alphabetically by task or group name.
|
| 478 |
+
# NOTE: we default here to false, because order matters for multi-level table printing a la mmlu.
|
| 479 |
+
# sorting here would mess that up
|
| 480 |
+
keys = sorted(keys)
|
| 481 |
+
for k in keys:
|
| 482 |
+
dic = result_dict[column][k]
|
| 483 |
+
version = result_dict["versions"].get(k, " N/A")
|
| 484 |
+
n = str(result_dict.get("n-shot", " ").get(k, " "))
|
| 485 |
+
higher_is_better = result_dict.get("higher_is_better", {}).get(k, {})
|
| 486 |
+
|
| 487 |
+
if "alias" in dic:
|
| 488 |
+
k = dic.pop("alias")
|
| 489 |
+
|
| 490 |
+
metric_items = dic.items()
|
| 491 |
+
metric_items = sorted(metric_items)
|
| 492 |
+
|
| 493 |
+
for (mf), v in metric_items:
|
| 494 |
+
m, _, f = mf.partition(",")
|
| 495 |
+
if m.endswith("_stderr"):
|
| 496 |
+
continue
|
| 497 |
+
|
| 498 |
+
hib = HIGHER_IS_BETTER_SYMBOLS.get(higher_is_better.get(m), "")
|
| 499 |
+
|
| 500 |
+
v = "%.4f" % v if isinstance(v, float) else v
|
| 501 |
+
if v == "" or v is None:
|
| 502 |
+
v = "N/A"
|
| 503 |
+
|
| 504 |
+
if m + "_stderr" + "," + f in dic:
|
| 505 |
+
# if dic[m + "_stderr" + "," + f] != []:
|
| 506 |
+
se = dic[m + "_stderr" + "," + f]
|
| 507 |
+
se = " N/A" if se == "N/A" or se == [] else "%.4f" % se
|
| 508 |
+
if v != []:
|
| 509 |
+
values.append([k, version, f, n, m, hib, v, "±", se])
|
| 510 |
+
else:
|
| 511 |
+
values.append([k, version, f, n, m, hib, v, "", ""])
|
| 512 |
+
# k = ""
|
| 513 |
+
# version = ""
|
| 514 |
+
md_writer.value_matrix = values
|
| 515 |
+
latex_writer.value_matrix = values
|
| 516 |
+
|
| 517 |
+
# todo: make latex table look good
|
| 518 |
+
# print(latex_writer.dumps())
|
| 519 |
+
|
| 520 |
+
return md_writer.dumps()
|
| 521 |
+
|
| 522 |
+
|
| 523 |
+
def positional_deprecated(fn):
|
| 524 |
+
"""
|
| 525 |
+
A decorator to nudge users into passing only keyword args (`kwargs`) to the
|
| 526 |
+
wrapped function, `fn`.
|
| 527 |
+
"""
|
| 528 |
+
|
| 529 |
+
@functools.wraps(fn)
|
| 530 |
+
def _wrapper(*args, **kwargs):
|
| 531 |
+
if len(args) != 1 if inspect.ismethod(fn) else 0:
|
| 532 |
+
print(f"WARNING: using {fn.__name__} with positional arguments is " "deprecated and will be disallowed in a future version of " "lmms-evaluation-harness!")
|
| 533 |
+
return fn(*args, **kwargs)
|
| 534 |
+
|
| 535 |
+
return _wrapper
|
| 536 |
+
|
| 537 |
+
|
| 538 |
+
@positional_deprecated
|
| 539 |
+
def find_test_root(start_path: pathlib.Path) -> pathlib.Path:
|
| 540 |
+
"""
|
| 541 |
+
Search upward in the directory tree to a maximum of three layers
|
| 542 |
+
to find and return the package root (containing the 'tests' folder)
|
| 543 |
+
"""
|
| 544 |
+
cur_path = start_path.resolve()
|
| 545 |
+
max_layers = 3
|
| 546 |
+
for _ in range(max_layers):
|
| 547 |
+
if (cur_path / "tests" / "test_version_stable.py").exists():
|
| 548 |
+
return cur_path
|
| 549 |
+
else:
|
| 550 |
+
cur_path = cur_path.parent.resolve()
|
| 551 |
+
raise FileNotFoundError(f"Unable to find package root within {max_layers} upwards" + f"of {start_path}")
|
| 552 |
+
|
| 553 |
+
|
| 554 |
+
@positional_deprecated
|
| 555 |
+
def run_task_tests(task_list: List[str]):
|
| 556 |
+
"""
|
| 557 |
+
Find the package root and run the tests for the given tasks
|
| 558 |
+
"""
|
| 559 |
+
import pytest
|
| 560 |
+
|
| 561 |
+
package_root = find_test_root(start_path=pathlib.Path(__file__))
|
| 562 |
+
task_string = " or ".join(task_list)
|
| 563 |
+
args = [
|
| 564 |
+
f"{package_root}/tests/test_version_stable.py",
|
| 565 |
+
f"--rootdir={package_root}",
|
| 566 |
+
"-k",
|
| 567 |
+
f"{task_string}",
|
| 568 |
+
]
|
| 569 |
+
sys.path.append(str(package_root))
|
| 570 |
+
pytest_return_val = pytest.main(args)
|
| 571 |
+
if pytest_return_val:
|
| 572 |
+
raise ValueError(f"Not all tests for the specified tasks ({task_list}) ran successfully! Error code: {pytest_return_val}")
|
| 573 |
+
|
| 574 |
+
|
| 575 |
+
def get_git_commit_hash():
|
| 576 |
+
"""
|
| 577 |
+
Gets the git commit hash of your current repo (if it exists).
|
| 578 |
+
Source: https://github.com/EleutherAI/gpt-neox/blob/b608043be541602170bfcfb8ec9bf85e8a0799e0/megatron/neox_arguments/neox_args.py#L42
|
| 579 |
+
"""
|
| 580 |
+
try:
|
| 581 |
+
git_hash = subprocess.check_output(["git", "describe", "--always"]).strip()
|
| 582 |
+
git_hash = git_hash.decode()
|
| 583 |
+
except subprocess.CalledProcessError or FileNotFoundError:
|
| 584 |
+
# FileNotFoundError occurs when git not installed on system
|
| 585 |
+
git_hash = None
|
| 586 |
+
return git_hash
|
| 587 |
+
|
| 588 |
+
|
| 589 |
+
def get_datetime_str(timezone="Asia/Singapore"):
|
| 590 |
+
"""
|
| 591 |
+
Gets the current datetime in UTC+8 timezone as a string.
|
| 592 |
+
"""
|
| 593 |
+
# Default: UTC+8 timezone
|
| 594 |
+
tz = pytz.timezone(timezone)
|
| 595 |
+
utc_now = datetime.datetime.now(datetime.timezone.utc)
|
| 596 |
+
local_time = utc_now.astimezone(tz)
|
| 597 |
+
return local_time.strftime("%Y%m%d_%H%M%S")
|
| 598 |
+
return local_time.strftime("%Y%m%d_%H%M%S")
|
| 599 |
+
|
| 600 |
+
|
| 601 |
+
def sanitize_long_string(s, max_length=40):
|
| 602 |
+
if len(s) > max_length:
|
| 603 |
+
return s[: max_length // 2] + "..." + s[-max_length // 2 :]
|
| 604 |
+
return s
|
| 605 |
+
|
| 606 |
+
|
| 607 |
+
def ignore_constructor(loader, node):
|
| 608 |
+
return node
|
| 609 |
+
|
| 610 |
+
|
| 611 |
+
def import_function(loader, node):
|
| 612 |
+
function_name = loader.construct_scalar(node)
|
| 613 |
+
yaml_path = os.path.dirname(loader.name)
|
| 614 |
+
|
| 615 |
+
*module_name, function_name = function_name.split(".")
|
| 616 |
+
if isinstance(module_name, list):
|
| 617 |
+
module_name = ".".join(module_name)
|
| 618 |
+
module_path = os.path.normpath(os.path.join(yaml_path, "{}.py".format(module_name)))
|
| 619 |
+
|
| 620 |
+
spec = importlib.util.spec_from_file_location(module_name, module_path)
|
| 621 |
+
module = importlib.util.module_from_spec(spec)
|
| 622 |
+
spec.loader.exec_module(module)
|
| 623 |
+
|
| 624 |
+
function = getattr(module, function_name)
|
| 625 |
+
return function
|
| 626 |
+
|
| 627 |
+
|
| 628 |
+
def load_yaml_config(yaml_path=None, yaml_config=None, yaml_dir=None, mode="full"):
|
| 629 |
+
if mode == "simple":
|
| 630 |
+
constructor_fn = ignore_constructor
|
| 631 |
+
elif mode == "full":
|
| 632 |
+
constructor_fn = import_function
|
| 633 |
+
|
| 634 |
+
# Add the import_function constructor to the YAML loader
|
| 635 |
+
yaml.add_constructor("!function", constructor_fn)
|
| 636 |
+
if yaml_config is None:
|
| 637 |
+
with open(yaml_path, "rb") as file:
|
| 638 |
+
yaml_config = yaml.full_load(file)
|
| 639 |
+
|
| 640 |
+
if yaml_dir is None:
|
| 641 |
+
yaml_dir = os.path.dirname(yaml_path)
|
| 642 |
+
|
| 643 |
+
assert yaml_dir is not None
|
| 644 |
+
assert yaml_config is not None
|
| 645 |
+
|
| 646 |
+
if "include" in yaml_config:
|
| 647 |
+
include_path = yaml_config["include"]
|
| 648 |
+
del yaml_config["include"]
|
| 649 |
+
|
| 650 |
+
if isinstance(include_path, str):
|
| 651 |
+
include_path = [include_path]
|
| 652 |
+
|
| 653 |
+
# Load from the last one first
|
| 654 |
+
include_path.reverse()
|
| 655 |
+
final_yaml_config = {}
|
| 656 |
+
for path in include_path:
|
| 657 |
+
# Assumes that path is a full path.
|
| 658 |
+
# If not found, assume the included yaml
|
| 659 |
+
# is in the same dir as the original yaml
|
| 660 |
+
if not os.path.isfile(path):
|
| 661 |
+
path = os.path.join(yaml_dir, path)
|
| 662 |
+
|
| 663 |
+
try:
|
| 664 |
+
included_yaml_config = load_yaml_config(yaml_path=path, mode=mode)
|
| 665 |
+
final_yaml_config.update(included_yaml_config)
|
| 666 |
+
except Exception as ex:
|
| 667 |
+
# If failed to load, ignore
|
| 668 |
+
raise ex
|
| 669 |
+
|
| 670 |
+
final_yaml_config.update(yaml_config)
|
| 671 |
+
return final_yaml_config
|
| 672 |
+
return yaml_config
|
| 673 |
+
|
| 674 |
+
|
| 675 |
+
def regex_replace(string, pattern, repl, count: int = 0):
|
| 676 |
+
"""Implements the `re.sub` function as a custom Jinja filter."""
|
| 677 |
+
return re.sub(pattern, repl, string, count=count)
|
| 678 |
+
|
| 679 |
+
|
| 680 |
+
env = Environment(loader=BaseLoader, undefined=StrictUndefined)
|
| 681 |
+
env.filters["regex_replace"] = regex_replace
|
| 682 |
+
|
| 683 |
+
|
| 684 |
+
def apply_template(template: str, doc: dict) -> str:
|
| 685 |
+
rtemplate = env.from_string(template)
|
| 686 |
+
return rtemplate.render(**doc)
|
| 687 |
+
|
| 688 |
+
|
| 689 |
+
def create_iterator(raw_iterator, rank, world_size, limit=None):
|
| 690 |
+
"""
|
| 691 |
+
Method for creating a (potentially) sliced and limited
|
| 692 |
+
iterator from a raw document iterator. Used for splitting data
|
| 693 |
+
among ranks in multigpu setting or only pulling a sample of documents
|
| 694 |
+
"""
|
| 695 |
+
return islice(raw_iterator, rank, limit, world_size)
|
| 696 |
+
|
| 697 |
+
|
| 698 |
+
def pad_and_concat(
|
| 699 |
+
max_length: int,
|
| 700 |
+
tensors: List[torch.Tensor],
|
| 701 |
+
padding_side: Literal["right", "left"] = "right",
|
| 702 |
+
):
|
| 703 |
+
"""
|
| 704 |
+
Method for padding a list of tensors given the maximum tensor
|
| 705 |
+
length in the batch. Used for batching inputs and continuations in
|
| 706 |
+
seq2seq models.
|
| 707 |
+
"""
|
| 708 |
+
assert padding_side == "left" or padding_side == "right", f"Unrecognized padding type: '{padding_side}' not 'left' or 'right'"
|
| 709 |
+
|
| 710 |
+
for i, tensor in enumerate(tensors):
|
| 711 |
+
if len(tensor.shape) == 2:
|
| 712 |
+
tensor = tensor.squeeze(0) # squeeze, in case passed [1, seq] size
|
| 713 |
+
tensor_len = tensor.shape[0]
|
| 714 |
+
if tensor_len < max_length:
|
| 715 |
+
if padding_side == "right":
|
| 716 |
+
# right-pad
|
| 717 |
+
tensors[i] = torch.cat(
|
| 718 |
+
[
|
| 719 |
+
tensor, # [seq]
|
| 720 |
+
torch.zeros(
|
| 721 |
+
max_length - tensor_len,
|
| 722 |
+
dtype=torch.long,
|
| 723 |
+
device=tensor.device,
|
| 724 |
+
), # [padding_length - seq]
|
| 725 |
+
],
|
| 726 |
+
dim=0,
|
| 727 |
+
).unsqueeze(0)
|
| 728 |
+
else:
|
| 729 |
+
# left-pad
|
| 730 |
+
tensors[i] = torch.cat(
|
| 731 |
+
[
|
| 732 |
+
torch.zeros(
|
| 733 |
+
max_length - tensor_len,
|
| 734 |
+
dtype=torch.long,
|
| 735 |
+
device=tensor.device,
|
| 736 |
+
), # [padding_length - seq]
|
| 737 |
+
tensor, # [seq]
|
| 738 |
+
],
|
| 739 |
+
dim=0,
|
| 740 |
+
).unsqueeze(0)
|
| 741 |
+
else:
|
| 742 |
+
tensors[i] = tensor.unsqueeze(0)
|
| 743 |
+
|
| 744 |
+
return torch.cat(tensors, dim=0)
|
| 745 |
+
|
| 746 |
+
|
| 747 |
+
def clear_torch_cache() -> None:
|
| 748 |
+
gc.collect()
|
| 749 |
+
torch.cuda.empty_cache()
|
| 750 |
+
|
| 751 |
+
|
| 752 |
+
def get_dtype(dtype: Union[str, torch.dtype]) -> torch.dtype:
|
| 753 |
+
"""Converts `dtype` from `str` to torch.dtype when possible. Does not use an instantiated HF AutoConfig"""
|
| 754 |
+
if isinstance(dtype, str) and dtype != "auto":
|
| 755 |
+
# Convert `str` args torch dtype: `float16` -> `torch.float16`
|
| 756 |
+
_torch_dtype = getattr(torch, dtype)
|
| 757 |
+
else:
|
| 758 |
+
_torch_dtype = dtype
|
| 759 |
+
return _torch_dtype
|
| 760 |
+
|
| 761 |
+
|
| 762 |
+
# Multi-token stopping criteria
|
| 763 |
+
class MultiTokenEOSCriteria(transformers.StoppingCriteria):
|
| 764 |
+
"""Criteria to stop on the specified multi-token sequence."""
|
| 765 |
+
|
| 766 |
+
def __init__(
|
| 767 |
+
self,
|
| 768 |
+
sequence: str,
|
| 769 |
+
tokenizer: transformers.PreTrainedTokenizer,
|
| 770 |
+
initial_decoder_input_length: int,
|
| 771 |
+
batch_size: int,
|
| 772 |
+
) -> None:
|
| 773 |
+
self.initial_decoder_input_length = initial_decoder_input_length
|
| 774 |
+
self.done_tracker = [False] * batch_size
|
| 775 |
+
self.sequence = sequence
|
| 776 |
+
self.sequence_ids = tokenizer.encode(sequence, add_special_tokens=False)
|
| 777 |
+
# we look back for 2 more tokens than it takes to encode our stop sequence
|
| 778 |
+
# because tokenizers suck, and a model might generate `['\n', '\n']` but our `sequence` is `['\n\n']`
|
| 779 |
+
# and we don't want to mistakenly not stop a generation because our
|
| 780 |
+
# (string) stop sequence was output in a different tokenization
|
| 781 |
+
|
| 782 |
+
# NOTE: there is a minor danger that this will end up looking back 2 tokens into the past, into the inputs to the model,
|
| 783 |
+
# and stopping generation immediately as a result. With only 2 extra tokens of lookback, this risk is minimized
|
| 784 |
+
self.sequence_id_len = len(self.sequence_ids) + 2
|
| 785 |
+
self.tokenizer = tokenizer
|
| 786 |
+
|
| 787 |
+
def __call__(self, input_ids, scores, **kwargs) -> bool:
|
| 788 |
+
# For efficiency, we compare the last n tokens where n is the number of tokens in the stop_sequence
|
| 789 |
+
lookback_ids_batch = input_ids[:, self.initial_decoder_input_length :][:, -self.sequence_id_len :]
|
| 790 |
+
|
| 791 |
+
lookback_tokens_batch = self.tokenizer.batch_decode(lookback_ids_batch)
|
| 792 |
+
for i, done in enumerate(self.done_tracker):
|
| 793 |
+
if not done:
|
| 794 |
+
self.done_tracker[i] = self.sequence in lookback_tokens_batch[i]
|
| 795 |
+
return False not in self.done_tracker
|
| 796 |
+
|
| 797 |
+
|
| 798 |
+
def stop_sequences_criteria(
|
| 799 |
+
tokenizer: transformers.PreTrainedTokenizer,
|
| 800 |
+
stop_sequences: List[str],
|
| 801 |
+
initial_decoder_input_length: int,
|
| 802 |
+
batch_size: int,
|
| 803 |
+
) -> transformers.StoppingCriteriaList:
|
| 804 |
+
return transformers.StoppingCriteriaList(
|
| 805 |
+
[
|
| 806 |
+
*[MultiTokenEOSCriteria(sequence, tokenizer, initial_decoder_input_length, batch_size) for sequence in stop_sequences],
|
| 807 |
+
]
|
| 808 |
+
)
|
| 809 |
+
|
| 810 |
+
|
| 811 |
+
# from more_itertools
|
| 812 |
+
def divide(iterable, n) -> List[Iterator]:
|
| 813 |
+
"""Divide the elements from *iterable* into *n* parts, maintaining
|
| 814 |
+
order.
|
| 815 |
+
|
| 816 |
+
>>> group_1, group_2 = divide(2, [1, 2, 3, 4, 5, 6])
|
| 817 |
+
>>> list(group_1)
|
| 818 |
+
[1, 2, 3]
|
| 819 |
+
>>> list(group_2)
|
| 820 |
+
[4, 5, 6]
|
| 821 |
+
|
| 822 |
+
If the length of *iterable* is not evenly divisible by *n*, then the
|
| 823 |
+
length of the returned iterables will not be identical:
|
| 824 |
+
|
| 825 |
+
>>> children = divide(3, [1, 2, 3, 4, 5, 6, 7])
|
| 826 |
+
>>> [list(c) for c in children]
|
| 827 |
+
[[1, 2, 3], [4, 5], [6, 7]]
|
| 828 |
+
|
| 829 |
+
If the length of the iterable is smaller than n, then the last returned
|
| 830 |
+
iterables will be empty:
|
| 831 |
+
|
| 832 |
+
>>> children = divide(5, [1, 2, 3])
|
| 833 |
+
>>> [list(c) for c in children]
|
| 834 |
+
[[1], [2], [3], [], []]
|
| 835 |
+
|
| 836 |
+
This function will exhaust the iterable before returning and may require
|
| 837 |
+
significant storage. If order is not important, see :func:`distribute`,
|
| 838 |
+
which does not first pull the iterable into memory.
|
| 839 |
+
|
| 840 |
+
"""
|
| 841 |
+
if n < 1:
|
| 842 |
+
raise ValueError("n must be at least 1")
|
| 843 |
+
|
| 844 |
+
try:
|
| 845 |
+
iterable[:0]
|
| 846 |
+
except TypeError:
|
| 847 |
+
seq = tuple(iterable)
|
| 848 |
+
else:
|
| 849 |
+
seq = iterable
|
| 850 |
+
|
| 851 |
+
q, r = divmod(len(seq), n)
|
| 852 |
+
|
| 853 |
+
ret = []
|
| 854 |
+
stop = 0
|
| 855 |
+
for i in range(1, n + 1):
|
| 856 |
+
start = stop
|
| 857 |
+
stop += q + 1 if i <= r else q
|
| 858 |
+
ret.append(iter(seq[start:stop]))
|
| 859 |
+
|
| 860 |
+
return ret
|
| 861 |
+
|
| 862 |
+
|
| 863 |
+
class Collator:
|
| 864 |
+
"""
|
| 865 |
+
A class for reordering and batching elements of an array.
|
| 866 |
+
|
| 867 |
+
This class allows for sorting an array based on a provided sorting function, grouping elements based on a grouping function, and generating batches from the sorted and grouped data.
|
| 868 |
+
"""
|
| 869 |
+
|
| 870 |
+
def __init__(
|
| 871 |
+
self,
|
| 872 |
+
arr: List,
|
| 873 |
+
sort_fn: Callable,
|
| 874 |
+
group_fn: Callable = lambda x: x[1],
|
| 875 |
+
grouping: bool = False,
|
| 876 |
+
) -> None:
|
| 877 |
+
self.grouping = grouping
|
| 878 |
+
self.fn = sort_fn
|
| 879 |
+
self.group_fn = lambda x: group_fn(x[1]) # first index are enumerated indices
|
| 880 |
+
self.reorder_indices: List = []
|
| 881 |
+
self.size = len(arr)
|
| 882 |
+
self.arr_with_indices: Iterable[Any] = tuple(enumerate(arr)) # [indices, (arr)]
|
| 883 |
+
if self.grouping is True:
|
| 884 |
+
self.group_by_index()
|
| 885 |
+
|
| 886 |
+
def group_by_index(self) -> None:
|
| 887 |
+
self.arr_with_indices = self.group(self.arr_with_indices, fn=self.group_fn, values=False)
|
| 888 |
+
|
| 889 |
+
def get_batched(self, n: int = 1, batch_fn: Optional[Callable] = None) -> Iterator:
|
| 890 |
+
"""
|
| 891 |
+
Generates and yields batches from the reordered array.
|
| 892 |
+
|
| 893 |
+
Parameters:
|
| 894 |
+
- n (int): The size of each batch. Defaults to 1.
|
| 895 |
+
- batch_fn (Optional[Callable[[int, Iterable], int]]): A function to determine the size of each batch. Defaults to None.
|
| 896 |
+
|
| 897 |
+
Yields:
|
| 898 |
+
Iterator: An iterator over batches of reordered elements.
|
| 899 |
+
"""
|
| 900 |
+
if self.grouping:
|
| 901 |
+
for (
|
| 902 |
+
key,
|
| 903 |
+
values,
|
| 904 |
+
) in self.arr_with_indices.items(): # type: ignore
|
| 905 |
+
values = self._reorder(values)
|
| 906 |
+
batch = self.get_chunks(values, n=n, fn=batch_fn)
|
| 907 |
+
yield from batch
|
| 908 |
+
else:
|
| 909 |
+
values = self._reorder(self.arr_with_indices) # type: ignore
|
| 910 |
+
batch = self.get_chunks(values, n=n, fn=batch_fn)
|
| 911 |
+
yield from batch
|
| 912 |
+
|
| 913 |
+
def _reorder(self, arr: Union[List, Tuple[Tuple[int, Any], ...]]) -> List:
|
| 914 |
+
"""
|
| 915 |
+
Reorders the elements in the array based on the sorting function.
|
| 916 |
+
|
| 917 |
+
Parameters:
|
| 918 |
+
- arr (Union[List, Tuple[Tuple[int, Any], ...]]): The array or iterable to be reordered.
|
| 919 |
+
|
| 920 |
+
Yields:
|
| 921 |
+
List: Yields reordered elements one by one.
|
| 922 |
+
"""
|
| 923 |
+
arr = sorted(arr, key=lambda x: self.fn(x[1]))
|
| 924 |
+
self.reorder_indices.extend([x[0] for x in arr])
|
| 925 |
+
yield from [x[1] for x in arr]
|
| 926 |
+
|
| 927 |
+
def get_original(self, newarr: List) -> List:
|
| 928 |
+
"""
|
| 929 |
+
Restores the original order of elements from the reordered list.
|
| 930 |
+
|
| 931 |
+
Parameters:
|
| 932 |
+
- newarr (List): The reordered array.
|
| 933 |
+
|
| 934 |
+
Returns:
|
| 935 |
+
List: The array with elements restored to their original order.
|
| 936 |
+
"""
|
| 937 |
+
res = [None] * self.size
|
| 938 |
+
cov = [False] * self.size
|
| 939 |
+
|
| 940 |
+
for ind, v in zip(self.reorder_indices, newarr):
|
| 941 |
+
res[ind] = v
|
| 942 |
+
cov[ind] = True
|
| 943 |
+
|
| 944 |
+
assert all(cov)
|
| 945 |
+
|
| 946 |
+
return res
|
| 947 |
+
|
| 948 |
+
def __len__(self):
|
| 949 |
+
return self.size
|
| 950 |
+
|
| 951 |
+
@staticmethod
|
| 952 |
+
def group(arr: Iterable, fn: Callable, values: bool = False) -> Iterable:
|
| 953 |
+
"""
|
| 954 |
+
Groups elements of an iterable based on a provided function.
|
| 955 |
+
|
| 956 |
+
Parameters:
|
| 957 |
+
- arr (Iterable): The iterable to be grouped.
|
| 958 |
+
- fn (Callable): The function to determine the grouping.
|
| 959 |
+
- values (bool): If True, returns the values of the group. Defaults to False.
|
| 960 |
+
|
| 961 |
+
Returns:
|
| 962 |
+
Iterable: An iterable of grouped elements.
|
| 963 |
+
"""
|
| 964 |
+
res = collections.defaultdict(list)
|
| 965 |
+
for ob in arr:
|
| 966 |
+
try:
|
| 967 |
+
hashable_dict = tuple(
|
| 968 |
+
(
|
| 969 |
+
key,
|
| 970 |
+
tuple(value) if isinstance(value, collections.abc.Iterable) else value,
|
| 971 |
+
)
|
| 972 |
+
for key, value in sorted(fn(ob).items())
|
| 973 |
+
)
|
| 974 |
+
res[hashable_dict].append(ob)
|
| 975 |
+
except TypeError:
|
| 976 |
+
res[fn(ob)].append(ob)
|
| 977 |
+
if not values:
|
| 978 |
+
return res
|
| 979 |
+
return res.values()
|
| 980 |
+
|
| 981 |
+
@staticmethod
|
| 982 |
+
def get_chunks(_iter, n: int = 0, fn=None):
|
| 983 |
+
"""
|
| 984 |
+
Divides an iterable into chunks of specified size or based on a given function.
|
| 985 |
+
Useful for batching
|
| 986 |
+
|
| 987 |
+
Parameters:
|
| 988 |
+
- iter: The input iterable to be divided into chunks.
|
| 989 |
+
- n: An integer representing the size of each chunk. Default is 0.
|
| 990 |
+
- fn: A function that takes the current index and the iterable as arguments and returns the size of the chunk. Default is None.
|
| 991 |
+
|
| 992 |
+
Returns:
|
| 993 |
+
An iterator that yields chunks of the input iterable.
|
| 994 |
+
|
| 995 |
+
Example usage:
|
| 996 |
+
```
|
| 997 |
+
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
|
| 998 |
+
for chunk in chunks(data, 3):
|
| 999 |
+
print(chunk)
|
| 1000 |
+
```
|
| 1001 |
+
Output:
|
| 1002 |
+
```
|
| 1003 |
+
[1, 2, 3]
|
| 1004 |
+
[4, 5, 6]
|
| 1005 |
+
[7, 8, 9]
|
| 1006 |
+
[10]
|
| 1007 |
+
```
|
| 1008 |
+
"""
|
| 1009 |
+
arr = []
|
| 1010 |
+
_iter = tuple(_iter)
|
| 1011 |
+
for i, x in enumerate(_iter):
|
| 1012 |
+
arr.append(x)
|
| 1013 |
+
if len(arr) == (fn(i, _iter) if fn else n):
|
| 1014 |
+
yield arr
|
| 1015 |
+
arr = []
|
| 1016 |
+
|
| 1017 |
+
if arr:
|
| 1018 |
+
yield arr
|
LaViDa/eval/pyproject.toml
ADDED
|
@@ -0,0 +1,161 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[tool.black]
|
| 2 |
+
line-length = 240
|
| 3 |
+
|
| 4 |
+
[build-system]
|
| 5 |
+
requires = ["setuptools>=42", "wheel", "setuptools_scm[tomli]>=6.3"]
|
| 6 |
+
build-backend = "setuptools.build_meta"
|
| 7 |
+
|
| 8 |
+
[project]
|
| 9 |
+
name = "lmms_eval"
|
| 10 |
+
version = "0.3.0"
|
| 11 |
+
authors = [
|
| 12 |
+
{ name = "LMMMs-Lab Evaluation Team", email = "lmms-lab@outlook.com" },
|
| 13 |
+
]
|
| 14 |
+
description = "A framework for evaluating large multi-modality language models"
|
| 15 |
+
readme = "README.md"
|
| 16 |
+
classifiers = [
|
| 17 |
+
"Programming Language :: Python :: 3",
|
| 18 |
+
"License :: OSI Approved :: MIT License",
|
| 19 |
+
"Operating System :: OS Independent",
|
| 20 |
+
]
|
| 21 |
+
requires-python = ">=3.8"
|
| 22 |
+
license = { text = "MIT" }
|
| 23 |
+
dependencies = [
|
| 24 |
+
"accelerate>=0.29.1",
|
| 25 |
+
"black==24.1.0",
|
| 26 |
+
"isort==5.13.2",
|
| 27 |
+
"datasets>=2.16.1",
|
| 28 |
+
"evaluate>=0.4.0",
|
| 29 |
+
"httpx==0.23.3",
|
| 30 |
+
"jsonlines",
|
| 31 |
+
"numexpr",
|
| 32 |
+
"peft>=0.2.0",
|
| 33 |
+
"pybind11>=2.6.2",
|
| 34 |
+
"pytablewriter",
|
| 35 |
+
"sacrebleu>=1.5.0",
|
| 36 |
+
"sqlitedict==2.1.0",
|
| 37 |
+
"timm",
|
| 38 |
+
"einops",
|
| 39 |
+
"ftfy",
|
| 40 |
+
"openai",
|
| 41 |
+
"opencv-python-headless",
|
| 42 |
+
"av",
|
| 43 |
+
"hf_transfer",
|
| 44 |
+
"nltk",
|
| 45 |
+
"sentencepiece",
|
| 46 |
+
"yt-dlp",
|
| 47 |
+
"pycocoevalcap",
|
| 48 |
+
"tqdm-multiprocess",
|
| 49 |
+
"transformers-stream-generator",
|
| 50 |
+
"zstandard",
|
| 51 |
+
"pillow",
|
| 52 |
+
"pyyaml",
|
| 53 |
+
"sympy",
|
| 54 |
+
"latex2sympy2",
|
| 55 |
+
"mpmath",
|
| 56 |
+
"Jinja2",
|
| 57 |
+
"openpyxl",
|
| 58 |
+
"loguru",
|
| 59 |
+
"hf_transfer",
|
| 60 |
+
"tenacity==8.3.0",
|
| 61 |
+
"wandb>=0.16.0",
|
| 62 |
+
"tiktoken",
|
| 63 |
+
"pre-commit",
|
| 64 |
+
"pydantic",
|
| 65 |
+
"packaging",
|
| 66 |
+
"decord",
|
| 67 |
+
"zss",
|
| 68 |
+
"protobuf",
|
| 69 |
+
"sentence_transformers",
|
| 70 |
+
"tokenizers"
|
| 71 |
+
]
|
| 72 |
+
|
| 73 |
+
[project.optional-dependencies]
|
| 74 |
+
audio = [
|
| 75 |
+
"more-itertools",
|
| 76 |
+
"editdistance",
|
| 77 |
+
"zhconv",
|
| 78 |
+
"librosa",
|
| 79 |
+
"soundfile"
|
| 80 |
+
]
|
| 81 |
+
metrics = [
|
| 82 |
+
"pywsd",
|
| 83 |
+
"spacy",
|
| 84 |
+
"anls",
|
| 85 |
+
"rouge",
|
| 86 |
+
"capture_metric",
|
| 87 |
+
"Levenshtein",
|
| 88 |
+
]
|
| 89 |
+
gemini = [
|
| 90 |
+
"google-generativeai",
|
| 91 |
+
]
|
| 92 |
+
reka = [
|
| 93 |
+
"httpx==0.23.3",
|
| 94 |
+
"reka-api",
|
| 95 |
+
]
|
| 96 |
+
qwen = [
|
| 97 |
+
"decord",
|
| 98 |
+
"qwen_vl_utils",
|
| 99 |
+
]
|
| 100 |
+
mmsearch = [
|
| 101 |
+
"playwright",
|
| 102 |
+
"requests",
|
| 103 |
+
"matplotlib",
|
| 104 |
+
"duckduckgo_search",
|
| 105 |
+
"langchain",
|
| 106 |
+
"langchain-community",
|
| 107 |
+
"beautifulsoup4",
|
| 108 |
+
"FlagEmbedding",
|
| 109 |
+
"rouge",
|
| 110 |
+
]
|
| 111 |
+
all = [
|
| 112 |
+
"gemini",
|
| 113 |
+
"reka",
|
| 114 |
+
"metrics",
|
| 115 |
+
"qwen",
|
| 116 |
+
"mmsearch"
|
| 117 |
+
]
|
| 118 |
+
|
| 119 |
+
[tool.setuptools.packages.find]
|
| 120 |
+
include = ["lmms_eval*"]
|
| 121 |
+
exclude = [
|
| 122 |
+
"assets*",
|
| 123 |
+
"benchmark*",
|
| 124 |
+
"docs",
|
| 125 |
+
"dist*",
|
| 126 |
+
"playground*",
|
| 127 |
+
"scripts*",
|
| 128 |
+
"tests*",
|
| 129 |
+
"checkpoints*",
|
| 130 |
+
"project_checkpoints*",
|
| 131 |
+
"debug_checkpoints*",
|
| 132 |
+
"mlx_configs*",
|
| 133 |
+
"wandb*",
|
| 134 |
+
"notebooks*",
|
| 135 |
+
"logs*",
|
| 136 |
+
]
|
| 137 |
+
|
| 138 |
+
[tool.wheel]
|
| 139 |
+
exclude = [
|
| 140 |
+
"assets*",
|
| 141 |
+
"benchmark*",
|
| 142 |
+
"docs",
|
| 143 |
+
"dist*",
|
| 144 |
+
"playground*",
|
| 145 |
+
"scripts*",
|
| 146 |
+
"tests*",
|
| 147 |
+
"checkpoints*",
|
| 148 |
+
"project_checkpoints*",
|
| 149 |
+
"debug_checkpoints*",
|
| 150 |
+
"mlx_configs*",
|
| 151 |
+
"wandb*",
|
| 152 |
+
"notebooks*",
|
| 153 |
+
"logs*",
|
| 154 |
+
]
|
| 155 |
+
|
| 156 |
+
[project.scripts]
|
| 157 |
+
lmms-eval = "lmms_eval.__main__:cli_evaluate"
|
| 158 |
+
|
| 159 |
+
[project.urls]
|
| 160 |
+
Homepage = "https://lmms-lab.github.io"
|
| 161 |
+
Repository = "https://github.com/EvolvingLMMs-Lab/lmms-eval"
|
LaViDa/eval/run_coco_dream.sh
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
LLADA_VISION_ENCODER="google/siglip-so400m-patch14-384"
|
| 2 |
+
set -x
|
| 3 |
+
export TASKS=${TASKS:-"mme"}
|
| 4 |
+
export CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7,8
|
| 5 |
+
export DEBUG_PRINT_IMAGE_RES=1
|
| 6 |
+
# max_new_tokens
|
| 7 |
+
accelerate launch --num_processes=8 --main_process_port=25511\
|
| 8 |
+
-m lmms_eval \
|
| 9 |
+
--model llava_dream \
|
| 10 |
+
--model_args pretrained=$1,conv_template=dream,model_name=llava_dream \
|
| 11 |
+
--tasks $TASKS \
|
| 12 |
+
--batch_size 1 \
|
| 13 |
+
--gen_kwargs alg=topk_margin,prefix_lm=True,step_ratio=0.5,schedule=shift,schedule__shift=0.33 \
|
| 14 |
+
--log_samples \
|
| 15 |
+
--log_samples_suffix llava_llada \
|
| 16 |
+
--output_path ./logs/ --verbosity=DEBUG \
|
| 17 |
+
${@:2} \
|
LaViDa/scripts/zero2_fused_adamw.json
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"fp16": {
|
| 3 |
+
"enabled": "auto",
|
| 4 |
+
"loss_scale": 0,
|
| 5 |
+
"loss_scale_window": 1000,
|
| 6 |
+
"initial_scale_power": 16,
|
| 7 |
+
"hysteresis": 2,
|
| 8 |
+
"min_loss_scale": 1
|
| 9 |
+
},
|
| 10 |
+
"bf16": {
|
| 11 |
+
"enabled": "auto"
|
| 12 |
+
},
|
| 13 |
+
"optimizer": {
|
| 14 |
+
"type": "AdamW",
|
| 15 |
+
"params": {
|
| 16 |
+
"lr": "auto",
|
| 17 |
+
"betas": "auto",
|
| 18 |
+
"eps": "auto",
|
| 19 |
+
"weight_decay": "auto"
|
| 20 |
+
}
|
| 21 |
+
},
|
| 22 |
+
"zero_optimization": {
|
| 23 |
+
"stage": 2,
|
| 24 |
+
"offload_optimizer": {
|
| 25 |
+
"device": "none",
|
| 26 |
+
"pin_memory": true
|
| 27 |
+
},
|
| 28 |
+
"allgather_partitions": true,
|
| 29 |
+
"allgather_bucket_size": 2e8,
|
| 30 |
+
"overlap_comm": true,
|
| 31 |
+
"reduce_scatter": true,
|
| 32 |
+
"reduce_bucket_size": 2e8,
|
| 33 |
+
"contiguous_gradients": true
|
| 34 |
+
},
|
| 35 |
+
"gradient_accumulation_steps": "auto",
|
| 36 |
+
"gradient_clipping": "auto",
|
| 37 |
+
"steps_per_print": 100,
|
| 38 |
+
"train_batch_size": "auto",
|
| 39 |
+
"train_micro_batch_size_per_gpu": "auto",
|
| 40 |
+
"wall_clock_breakdown": false
|
| 41 |
+
}
|
rl_lladav_video_reversal_wo_t_reward/results/all_rewards.csv
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
step,acc_reward,t_grpo_reward,unbiased_reward,response_length
|
| 2 |
+
1,0.5449,0.0000,0.1125,127.96
|
| 3 |
+
step,acc_reward,t_grpo_reward,unbiased_reward,response_length
|
| 4 |
+
1,0.5449,0.0000,0.1125,127.96
|
| 5 |
+
2,0.5664,0.0000,0.1125,128.00
|
| 6 |
+
2,0.5664,0.0000,0.1125,128.00
|
| 7 |
+
3,0.5000,0.0000,0.1031,128.00
|
| 8 |
+
3,0.5000,0.0000,0.1031,128.00
|
| 9 |
+
4,0.5410,0.0000,0.1500,127.98
|
| 10 |
+
4,0.5410,0.0000,0.1500,127.98
|
| 11 |
+
5,0.4922,0.0000,0.0937,127.99
|
| 12 |
+
5,0.4922,0.0000,0.0937,127.99
|
| 13 |
+
6,0.5215,0.0000,0.1219,127.99
|
| 14 |
+
6,0.5215,0.0000,0.1219,127.99
|
| 15 |
+
7,0.4805,0.0000,0.0937,128.00
|
| 16 |
+
7,0.4805,0.0000,0.0937,128.00
|
| 17 |
+
8,0.4297,0.0000,0.1219,128.00
|
| 18 |
+
8,0.4297,0.0000,0.1219,128.00
|
| 19 |
+
9,0.5234,0.0000,0.1219,128.00
|
| 20 |
+
9,0.5234,0.0000,0.1219,128.00
|
| 21 |
+
10,0.5098,0.0000,0.1500,128.00
|
| 22 |
+
10,0.5098,0.0000,0.1500,128.00
|
| 23 |
+
11,0.5000,0.0000,0.1031,127.99
|
| 24 |
+
11,0.5000,0.0000,0.1031,127.99
|
| 25 |
+
12,0.5859,0.0000,0.1031,127.99
|
| 26 |
+
12,0.5859,0.0000,0.1031,127.99
|
| 27 |
+
13,0.4883,0.0000,0.0844,127.99
|
| 28 |
+
13,0.4883,0.0000,0.0844,127.99
|
| 29 |
+
14,0.5430,0.0000,0.1125,127.99
|
| 30 |
+
14,0.5430,0.0000,0.1125,127.99
|
| 31 |
+
15,0.5391,0.0000,0.1594,128.00
|
| 32 |
+
15,0.5391,0.0000,0.1594,128.00
|
| 33 |
+
16,0.4961,0.0000,0.1219,127.99
|
| 34 |
+
16,0.4961,0.0000,0.1219,127.99
|
| 35 |
+
17,0.5332,0.0000,0.0844,128.00
|
| 36 |
+
17,0.5332,0.0000,0.0844,128.00
|
| 37 |
+
18,0.5723,0.0000,0.1406,128.00
|
| 38 |
+
18,0.5723,0.0000,0.1406,128.00
|
| 39 |
+
19,0.6250,0.0000,0.1781,128.00
|
| 40 |
+
19,0.6250,0.0000,0.1781,128.00
|
| 41 |
+
20,0.5312,0.0000,0.1031,127.98
|
| 42 |
+
20,0.5312,0.0000,0.1031,127.98
|
| 43 |
+
21,0.5703,0.0000,0.1219,128.00
|
| 44 |
+
21,0.5703,0.0000,0.1219,128.00
|
| 45 |
+
22,0.4238,0.0000,0.1500,127.99
|
| 46 |
+
22,0.4238,0.0000,0.1500,127.99
|
| 47 |
+
23,0.5859,0.0000,0.0937,128.00
|
| 48 |
+
23,0.5859,0.0000,0.0937,128.00
|
| 49 |
+
24,0.6133,0.0000,0.1219,127.99
|
| 50 |
+
24,0.6133,0.0000,0.1219,127.99
|
| 51 |
+
25,0.5371,0.0000,0.0844,128.00
|
| 52 |
+
25,0.5371,0.0000,0.0844,128.00
|
| 53 |
+
26,0.5059,0.0000,0.1406,127.99
|
| 54 |
+
26,0.5059,0.0000,0.1406,127.99
|
| 55 |
+
27,0.6055,0.0000,0.1219,128.00
|
| 56 |
+
27,0.6055,0.0000,0.1219,128.00
|
| 57 |
+
28,0.5703,0.0000,0.1219,128.00
|
| 58 |
+
28,0.5703,0.0000,0.1219,128.00
|
| 59 |
+
29,0.6035,0.0000,0.1312,127.99
|
| 60 |
+
29,0.6035,0.0000,0.1312,127.99
|
| 61 |
+
30,0.6270,0.0000,0.1594,127.99
|
| 62 |
+
30,0.6270,0.0000,0.1594,127.99
|
rl_lladav_video_reversal_wo_t_reward/results/results-eval-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train.txt
ADDED
|
File without changes
|
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step11.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
train step: 11 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5000 avg length: 127.99
|
| 2 |
+
train step: 11 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5000 avg length: 127.99
|
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step23.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
train step: 23 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5859 avg length: 128.00
|
| 2 |
+
train step: 23 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5859 avg length: 128.00
|
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step6.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
train step: 6 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5215 avg length: 127.99
|
| 2 |
+
train step: 6 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5215 avg length: 127.99
|
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step7.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
train step: 7 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.4805 avg length: 128.00
|
| 2 |
+
train step: 7 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.4805 avg length: 128.00
|
rl_lladav_video_reversal_wo_t_reward/results/results-rl-...rl_lladav_video_reversal_wo_t_reward.ckpt.optimized-video_reversal_rl_train-step9.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
train step: 9 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5234 avg length: 128.00
|
| 2 |
+
train step: 9 remasking_strategy: low_confidence_dynamic block_size: 32 acc: 0.5234 avg length: 128.00
|