File size: 18,367 Bytes
f44837e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | [2026-07-09 11:28:18,876][model][INFO] - Setting pad_token as eos token: <|end_of_text|>
[2026-07-09 11:29:10,500][trainer][INFO] - GradDiff Trainer loaded, output_dir: ./saves/unlearn/multilingual/Llama-3.1-8B/books.all/chapters.all/GradDiff.NLL/th
[2026-07-09 13:45:41,661][lm_eval.models.huggingface][WARNING] - `pretrained` model kwarg is not of type `str`. Many other model arguments may be ignored. Please do not launch via accelerate or use `parallelize=True` if passing an existing model this way.
[2026-07-09 13:45:43,567][lm_eval.models.huggingface][WARNING] - Passed an already-initialized model through `pretrained`, assuming single-process call to evaluate() or custom distributed integration
[2026-07-09 13:45:43,569][lm_eval.evaluator][INFO] - Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
[2026-07-09 13:45:43,569][lm_eval.evaluator][INFO] - Using pre-initialized model
[2026-07-09 13:47:26,456][lm_eval.evaluator_utils][INFO] - Selected tasks:
[2026-07-09 13:47:26,456][lm_eval.evaluator_utils][INFO] - Group: mmlu
[2026-07-09 13:47:26,457][lm_eval.evaluator_utils][INFO] - Group: mmlu_stem
[2026-07-09 13:47:26,457][lm_eval.evaluator_utils][INFO] - Task: mmlu_abstract_algebra (mmlu/default/mmlu_abstract_algebra.yaml)
[2026-07-09 13:47:26,458][lm_eval.evaluator_utils][INFO] - Task: mmlu_anatomy (mmlu/default/mmlu_anatomy.yaml)
[2026-07-09 13:47:26,458][lm_eval.evaluator_utils][INFO] - Task: mmlu_astronomy (mmlu/default/mmlu_astronomy.yaml)
[2026-07-09 13:47:26,458][lm_eval.evaluator_utils][INFO] - Task: mmlu_college_biology (mmlu/default/mmlu_college_biology.yaml)
[2026-07-09 13:47:26,458][lm_eval.evaluator_utils][INFO] - Task: mmlu_college_chemistry (mmlu/default/mmlu_college_chemistry.yaml)
[2026-07-09 13:47:26,458][lm_eval.evaluator_utils][INFO] - Task: mmlu_college_computer_science (mmlu/default/mmlu_college_computer_science.yaml)
[2026-07-09 13:47:26,459][lm_eval.evaluator_utils][INFO] - Task: mmlu_college_mathematics (mmlu/default/mmlu_college_mathematics.yaml)
[2026-07-09 13:47:26,459][lm_eval.evaluator_utils][INFO] - Task: mmlu_college_physics (mmlu/default/mmlu_college_physics.yaml)
[2026-07-09 13:47:26,459][lm_eval.evaluator_utils][INFO] - Task: mmlu_computer_security (mmlu/default/mmlu_computer_security.yaml)
[2026-07-09 13:47:26,459][lm_eval.evaluator_utils][INFO] - Task: mmlu_conceptual_physics (mmlu/default/mmlu_conceptual_physics.yaml)
[2026-07-09 13:47:26,460][lm_eval.evaluator_utils][INFO] - Task: mmlu_electrical_engineering (mmlu/default/mmlu_electrical_engineering.yaml)
[2026-07-09 13:47:26,460][lm_eval.evaluator_utils][INFO] - Task: mmlu_elementary_mathematics (mmlu/default/mmlu_elementary_mathematics.yaml)
[2026-07-09 13:47:26,460][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_biology (mmlu/default/mmlu_high_school_biology.yaml)
[2026-07-09 13:47:26,460][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_chemistry (mmlu/default/mmlu_high_school_chemistry.yaml)
[2026-07-09 13:47:26,461][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_computer_science (mmlu/default/mmlu_high_school_computer_science.yaml)
[2026-07-09 13:47:26,461][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_mathematics (mmlu/default/mmlu_high_school_mathematics.yaml)
[2026-07-09 13:47:26,461][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_physics (mmlu/default/mmlu_high_school_physics.yaml)
[2026-07-09 13:47:26,461][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_statistics (mmlu/default/mmlu_high_school_statistics.yaml)
[2026-07-09 13:47:26,462][lm_eval.evaluator_utils][INFO] - Task: mmlu_machine_learning (mmlu/default/mmlu_machine_learning.yaml)
[2026-07-09 13:47:26,462][lm_eval.evaluator_utils][INFO] - Group: mmlu_other
[2026-07-09 13:47:26,462][lm_eval.evaluator_utils][INFO] - Task: mmlu_business_ethics (mmlu/default/mmlu_business_ethics.yaml)
[2026-07-09 13:47:26,462][lm_eval.evaluator_utils][INFO] - Task: mmlu_clinical_knowledge (mmlu/default/mmlu_clinical_knowledge.yaml)
[2026-07-09 13:47:26,462][lm_eval.evaluator_utils][INFO] - Task: mmlu_college_medicine (mmlu/default/mmlu_college_medicine.yaml)
[2026-07-09 13:47:26,463][lm_eval.evaluator_utils][INFO] - Task: mmlu_global_facts (mmlu/default/mmlu_global_facts.yaml)
[2026-07-09 13:47:26,463][lm_eval.evaluator_utils][INFO] - Task: mmlu_human_aging (mmlu/default/mmlu_human_aging.yaml)
[2026-07-09 13:47:26,463][lm_eval.evaluator_utils][INFO] - Task: mmlu_management (mmlu/default/mmlu_management.yaml)
[2026-07-09 13:47:26,463][lm_eval.evaluator_utils][INFO] - Task: mmlu_marketing (mmlu/default/mmlu_marketing.yaml)
[2026-07-09 13:47:26,463][lm_eval.evaluator_utils][INFO] - Task: mmlu_medical_genetics (mmlu/default/mmlu_medical_genetics.yaml)
[2026-07-09 13:47:26,464][lm_eval.evaluator_utils][INFO] - Task: mmlu_miscellaneous (mmlu/default/mmlu_miscellaneous.yaml)
[2026-07-09 13:47:26,464][lm_eval.evaluator_utils][INFO] - Task: mmlu_nutrition (mmlu/default/mmlu_nutrition.yaml)
[2026-07-09 13:47:26,464][lm_eval.evaluator_utils][INFO] - Task: mmlu_professional_accounting (mmlu/default/mmlu_professional_accounting.yaml)
[2026-07-09 13:47:26,464][lm_eval.evaluator_utils][INFO] - Task: mmlu_professional_medicine (mmlu/default/mmlu_professional_medicine.yaml)
[2026-07-09 13:47:26,464][lm_eval.evaluator_utils][INFO] - Task: mmlu_virology (mmlu/default/mmlu_virology.yaml)
[2026-07-09 13:47:26,465][lm_eval.evaluator_utils][INFO] - Group: mmlu_social_sciences
[2026-07-09 13:47:26,465][lm_eval.evaluator_utils][INFO] - Task: mmlu_econometrics (mmlu/default/mmlu_econometrics.yaml)
[2026-07-09 13:47:26,465][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_geography (mmlu/default/mmlu_high_school_geography.yaml)
[2026-07-09 13:47:26,465][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_government_and_politics (mmlu/default/mmlu_high_school_government_and_politics.yaml)
[2026-07-09 13:47:26,466][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_macroeconomics (mmlu/default/mmlu_high_school_macroeconomics.yaml)
[2026-07-09 13:47:26,466][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_microeconomics (mmlu/default/mmlu_high_school_microeconomics.yaml)
[2026-07-09 13:47:26,466][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_psychology (mmlu/default/mmlu_high_school_psychology.yaml)
[2026-07-09 13:47:26,466][lm_eval.evaluator_utils][INFO] - Task: mmlu_human_sexuality (mmlu/default/mmlu_human_sexuality.yaml)
[2026-07-09 13:47:26,466][lm_eval.evaluator_utils][INFO] - Task: mmlu_professional_psychology (mmlu/default/mmlu_professional_psychology.yaml)
[2026-07-09 13:47:26,467][lm_eval.evaluator_utils][INFO] - Task: mmlu_public_relations (mmlu/default/mmlu_public_relations.yaml)
[2026-07-09 13:47:26,467][lm_eval.evaluator_utils][INFO] - Task: mmlu_security_studies (mmlu/default/mmlu_security_studies.yaml)
[2026-07-09 13:47:26,467][lm_eval.evaluator_utils][INFO] - Task: mmlu_sociology (mmlu/default/mmlu_sociology.yaml)
[2026-07-09 13:47:26,467][lm_eval.evaluator_utils][INFO] - Task: mmlu_us_foreign_policy (mmlu/default/mmlu_us_foreign_policy.yaml)
[2026-07-09 13:47:26,468][lm_eval.evaluator_utils][INFO] - Group: mmlu_humanities
[2026-07-09 13:47:26,468][lm_eval.evaluator_utils][INFO] - Task: mmlu_formal_logic (mmlu/default/mmlu_formal_logic.yaml)
[2026-07-09 13:47:26,468][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_european_history (mmlu/default/mmlu_high_school_european_history.yaml)
[2026-07-09 13:47:26,468][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_us_history (mmlu/default/mmlu_high_school_us_history.yaml)
[2026-07-09 13:47:26,468][lm_eval.evaluator_utils][INFO] - Task: mmlu_high_school_world_history (mmlu/default/mmlu_high_school_world_history.yaml)
[2026-07-09 13:47:26,469][lm_eval.evaluator_utils][INFO] - Task: mmlu_international_law (mmlu/default/mmlu_international_law.yaml)
[2026-07-09 13:47:26,469][lm_eval.evaluator_utils][INFO] - Task: mmlu_jurisprudence (mmlu/default/mmlu_jurisprudence.yaml)
[2026-07-09 13:47:26,469][lm_eval.evaluator_utils][INFO] - Task: mmlu_logical_fallacies (mmlu/default/mmlu_logical_fallacies.yaml)
[2026-07-09 13:47:26,469][lm_eval.evaluator_utils][INFO] - Task: mmlu_moral_disputes (mmlu/default/mmlu_moral_disputes.yaml)
[2026-07-09 13:47:26,470][lm_eval.evaluator_utils][INFO] - Task: mmlu_moral_scenarios (mmlu/default/mmlu_moral_scenarios.yaml)
[2026-07-09 13:47:26,470][lm_eval.evaluator_utils][INFO] - Task: mmlu_philosophy (mmlu/default/mmlu_philosophy.yaml)
[2026-07-09 13:47:26,470][lm_eval.evaluator_utils][INFO] - Task: mmlu_prehistory (mmlu/default/mmlu_prehistory.yaml)
[2026-07-09 13:47:26,470][lm_eval.evaluator_utils][INFO] - Task: mmlu_professional_law (mmlu/default/mmlu_professional_law.yaml)
[2026-07-09 13:47:26,470][lm_eval.evaluator_utils][INFO] - Task: mmlu_world_religions (mmlu/default/mmlu_world_religions.yaml)
[2026-07-09 13:47:26,471][lm_eval.evaluator_utils][INFO] - Group: multilingual_unlearning
[2026-07-09 13:47:26,471][lm_eval.evaluator_utils][INFO] - Task: multilingual_unlearning_harry_potter_en (multilingual_unlearning/multilingual_unlearning_harry_potter_en.yaml)
[2026-07-09 13:47:26,471][lm_eval.evaluator_utils][INFO] - Task: multilingual_unlearning_harry_potter_fr (multilingual_unlearning/multilingual_unlearning_harry_potter_fr.yaml)
[2026-07-09 13:47:26,471][lm_eval.evaluator_utils][INFO] - Task: multilingual_unlearning_harry_potter_ja (multilingual_unlearning/multilingual_unlearning_harry_potter_ja.yaml)
[2026-07-09 13:47:26,471][lm_eval.evaluator_utils][INFO] - Task: multilingual_unlearning_harry_potter_vi (multilingual_unlearning/multilingual_unlearning_harry_potter_vi.yaml)
[2026-07-09 13:47:26,472][lm_eval.evaluator_utils][INFO] - Task: multilingual_unlearning_harry_potter_zh (multilingual_unlearning/multilingual_unlearning_harry_potter_zh.yaml)
[2026-07-09 13:47:26,472][lm_eval.evaluator_utils][INFO] - Task: multilingual_unlearning_harry_potter_ko (multilingual_unlearning/multilingual_unlearning_harry_potter_ko.yaml)
[2026-07-09 13:47:26,475][lm_eval.evaluator_utils][INFO] - Task: multilingual_unlearning_harry_potter_es (multilingual_unlearning/multilingual_unlearning_harry_potter_es.yaml)
[2026-07-09 13:47:26,475][lm_eval.evaluator_utils][INFO] - Task: multilingual_unlearning_harry_potter_de (multilingual_unlearning/multilingual_unlearning_harry_potter_de.yaml)
[2026-07-09 13:47:26,476][lm_eval.evaluator_utils][INFO] - Task: multilingual_unlearning_harry_potter_th (multilingual_unlearning/multilingual_unlearning_harry_potter_th.yaml)
[2026-07-09 13:47:26,481][lm_eval.api.task][INFO] - Building contexts for multilingual_unlearning_harry_potter_en on rank 0...
[2026-07-09 13:47:33,414][lm_eval.api.task][INFO] - Building contexts for multilingual_unlearning_harry_potter_fr on rank 0...
[2026-07-09 13:47:39,916][lm_eval.api.task][INFO] - Building contexts for multilingual_unlearning_harry_potter_ja on rank 0...
[2026-07-09 13:47:46,090][lm_eval.api.task][INFO] - Building contexts for multilingual_unlearning_harry_potter_vi on rank 0...
[2026-07-09 13:47:52,248][lm_eval.api.task][INFO] - Building contexts for multilingual_unlearning_harry_potter_zh on rank 0...
[2026-07-09 13:47:58,837][lm_eval.api.task][INFO] - Building contexts for multilingual_unlearning_harry_potter_ko on rank 0...
[2026-07-09 13:48:05,598][lm_eval.api.task][INFO] - Building contexts for multilingual_unlearning_harry_potter_es on rank 0...
[2026-07-09 13:48:11,717][lm_eval.api.task][INFO] - Building contexts for multilingual_unlearning_harry_potter_de on rank 0...
[2026-07-09 13:48:18,293][lm_eval.api.task][INFO] - Building contexts for multilingual_unlearning_harry_potter_th on rank 0...
[2026-07-09 13:48:24,509][lm_eval.api.task][INFO] - Building contexts for mmlu_abstract_algebra on rank 0...
[2026-07-09 13:48:24,615][lm_eval.api.task][INFO] - Building contexts for mmlu_anatomy on rank 0...
[2026-07-09 13:48:24,757][lm_eval.api.task][INFO] - Building contexts for mmlu_astronomy on rank 0...
[2026-07-09 13:48:24,918][lm_eval.api.task][INFO] - Building contexts for mmlu_college_biology on rank 0...
[2026-07-09 13:48:25,071][lm_eval.api.task][INFO] - Building contexts for mmlu_college_chemistry on rank 0...
[2026-07-09 13:48:25,175][lm_eval.api.task][INFO] - Building contexts for mmlu_college_computer_science on rank 0...
[2026-07-09 13:48:25,284][lm_eval.api.task][INFO] - Building contexts for mmlu_college_mathematics on rank 0...
[2026-07-09 13:48:25,391][lm_eval.api.task][INFO] - Building contexts for mmlu_college_physics on rank 0...
[2026-07-09 13:48:25,497][lm_eval.api.task][INFO] - Building contexts for mmlu_computer_security on rank 0...
[2026-07-09 13:48:25,603][lm_eval.api.task][INFO] - Building contexts for mmlu_conceptual_physics on rank 0...
[2026-07-09 13:48:25,847][lm_eval.api.task][INFO] - Building contexts for mmlu_electrical_engineering on rank 0...
[2026-07-09 13:48:25,997][lm_eval.api.task][INFO] - Building contexts for mmlu_elementary_mathematics on rank 0...
[2026-07-09 13:48:26,393][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_biology on rank 0...
[2026-07-09 13:48:26,712][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_chemistry on rank 0...
[2026-07-09 13:48:26,922][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_computer_science on rank 0...
[2026-07-09 13:48:27,026][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_mathematics on rank 0...
[2026-07-09 13:48:27,307][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_physics on rank 0...
[2026-07-09 13:48:27,464][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_statistics on rank 0...
[2026-07-09 13:48:27,687][lm_eval.api.task][INFO] - Building contexts for mmlu_machine_learning on rank 0...
[2026-07-09 13:48:27,805][lm_eval.api.task][INFO] - Building contexts for mmlu_business_ethics on rank 0...
[2026-07-09 13:48:27,910][lm_eval.api.task][INFO] - Building contexts for mmlu_clinical_knowledge on rank 0...
[2026-07-09 13:48:28,190][lm_eval.api.task][INFO] - Building contexts for mmlu_college_medicine on rank 0...
[2026-07-09 13:48:28,371][lm_eval.api.task][INFO] - Building contexts for mmlu_global_facts on rank 0...
[2026-07-09 13:48:28,476][lm_eval.api.task][INFO] - Building contexts for mmlu_human_aging on rank 0...
[2026-07-09 13:48:28,705][lm_eval.api.task][INFO] - Building contexts for mmlu_management on rank 0...
[2026-07-09 13:48:28,815][lm_eval.api.task][INFO] - Building contexts for mmlu_marketing on rank 0...
[2026-07-09 13:48:29,067][lm_eval.api.task][INFO] - Building contexts for mmlu_medical_genetics on rank 0...
[2026-07-09 13:48:29,172][lm_eval.api.task][INFO] - Building contexts for mmlu_miscellaneous on rank 0...
[2026-07-09 13:48:29,984][lm_eval.api.task][INFO] - Building contexts for mmlu_nutrition on rank 0...
[2026-07-09 13:48:30,303][lm_eval.api.task][INFO] - Building contexts for mmlu_professional_accounting on rank 0...
[2026-07-09 13:48:30,599][lm_eval.api.task][INFO] - Building contexts for mmlu_professional_medicine on rank 0...
[2026-07-09 13:48:30,883][lm_eval.api.task][INFO] - Building contexts for mmlu_virology on rank 0...
[2026-07-09 13:48:31,057][lm_eval.api.task][INFO] - Building contexts for mmlu_econometrics on rank 0...
[2026-07-09 13:48:31,180][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_geography on rank 0...
[2026-07-09 13:48:31,395][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_government_and_politics on rank 0...
[2026-07-09 13:48:31,596][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_macroeconomics on rank 0...
[2026-07-09 13:48:31,998][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_microeconomics on rank 0...
[2026-07-09 13:48:32,246][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_psychology on rank 0...
[2026-07-09 13:48:32,814][lm_eval.api.task][INFO] - Building contexts for mmlu_human_sexuality on rank 0...
[2026-07-09 13:48:32,954][lm_eval.api.task][INFO] - Building contexts for mmlu_professional_psychology on rank 0...
[2026-07-09 13:48:33,586][lm_eval.api.task][INFO] - Building contexts for mmlu_public_relations on rank 0...
[2026-07-09 13:48:33,701][lm_eval.api.task][INFO] - Building contexts for mmlu_security_studies on rank 0...
[2026-07-09 13:48:33,955][lm_eval.api.task][INFO] - Building contexts for mmlu_sociology on rank 0...
[2026-07-09 13:48:34,165][lm_eval.api.task][INFO] - Building contexts for mmlu_us_foreign_policy on rank 0...
[2026-07-09 13:48:34,270][lm_eval.api.task][INFO] - Building contexts for mmlu_formal_logic on rank 0...
[2026-07-09 13:48:34,404][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_european_history on rank 0...
[2026-07-09 13:48:34,575][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_us_history on rank 0...
[2026-07-09 13:48:34,786][lm_eval.api.task][INFO] - Building contexts for mmlu_high_school_world_history on rank 0...
[2026-07-09 13:48:35,032][lm_eval.api.task][INFO] - Building contexts for mmlu_international_law on rank 0...
[2026-07-09 13:48:35,157][lm_eval.api.task][INFO] - Building contexts for mmlu_jurisprudence on rank 0...
[2026-07-09 13:48:35,268][lm_eval.api.task][INFO] - Building contexts for mmlu_logical_fallacies on rank 0...
[2026-07-09 13:48:35,436][lm_eval.api.task][INFO] - Building contexts for mmlu_moral_disputes on rank 0...
[2026-07-09 13:48:35,793][lm_eval.api.task][INFO] - Building contexts for mmlu_moral_scenarios on rank 0...
[2026-07-09 13:48:36,717][lm_eval.api.task][INFO] - Building contexts for mmlu_philosophy on rank 0...
[2026-07-09 13:48:37,034][lm_eval.api.task][INFO] - Building contexts for mmlu_prehistory on rank 0...
[2026-07-09 13:48:37,368][lm_eval.api.task][INFO] - Building contexts for mmlu_professional_law on rank 0...
[2026-07-09 13:48:38,940][lm_eval.api.task][INFO] - Building contexts for mmlu_world_religions on rank 0...
[2026-07-09 13:48:39,116][lm_eval.evaluator][INFO] - Running loglikelihood requests
|