Spaces:

---

Intel

/

low_bit_open_llm_leaderboard

like 222

Running

Pipeline Failure Report

Model: google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant
Quantization Scheme: INT4 (W4A16)
Failed Phase: evaluate
Run ID: gemma-4-E2B-it-qat-q4_0-unquantized-assistant-AutoRound-W4A16-RTN
Error Category: model_author_unsupported_arch

Full Error Log

2026-07-17:15:57:05 INFO     [api.task:312] Building contexts for mmlu_moral_scenarios on rank 0...

  0%|          | 0/895 [00:00<?, ?it/s]
100%|██████████| 895/895 [00:00<00:00, 1437.29it/s]
2026-07-17:15:57:06 INFO     [api.task:312] Building contexts for mmlu_philosophy on rank 0...

  0%|          | 0/311 [00:00<?, ?it/s]
100%|██████████| 311/311 [00:00<00:00, 434.61it/s]
2026-07-17:15:57:07 INFO     [api.task:312] Building contexts for mmlu_prehistory on rank 0...

  0%|          | 0/324 [00:00<?, ?it/s]
100%|██████████| 324/324 [00:00<00:00, 1456.06it/s]
2026-07-17:15:57:07 INFO     [api.task:312] Building contexts for mmlu_professional_law on rank 0...

  0%|          | 0/1534 [00:00<?, ?it/s]
100%|██████████| 1534/1534 [00:01<00:00, 1487.98it/s]
2026-07-17:15:57:08 INFO     [api.task:312] Building contexts for mmlu_world_religions on rank 0...

  0%|          | 0/171 [00:00<?, ?it/s]
100%|██████████| 171/171 [00:00<00:00, 1407.33it/s]
2026-07-17:15:57:08 INFO     [api.task:312] Building contexts for hellaswag on rank 0...

  0%|          | 0/10042 [00:00<?, ?it/s]
100%|██████████| 10042/10042 [00:02<00:00, 3514.75it/s]
2026-07-17:15:57:11 INFO     [evaluator:585] Running loglikelihood requests

Tokenizing inputs:   0%|          | 0/100012 [00:00<?, ?it/s]
Tokenizing inputs: 100%|██████████| 100012/100012 [00:18<00:00, 5273.62it/s]

Running loglikelihood requests:   0%|          | 0/100012 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Traceback (most recent call last):
  File "/root/.venv/bin/lm_eval", line 10, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/__main__.py", line 10, in cli_evaluate
    parser.execute(args)
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/_cli/harness.py", line 60, in execute
    args.func(args)
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/_cli/run.py", line 391, in _execute
    results = simple_evaluate(
              ^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/utils.py", line 575, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/evaluator.py", line 358, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/utils.py", line 575, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/evaluator.py", line 596, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1227, in loglikelihood
    return super().loglikelihood(requests, disable_tqdm=disable_tqdm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/api/model.py", line 446, in loglikelihood
    return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1402, in _loglikelihood_tokens
    for chunk in chunks:
                 ^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/utils.py", line 315, in get_batched
    yield from batch
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/utils.py", line 492, in get_chunks
    if len(arr) == (fn(i, _iter) if fn else n):
                    ^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1328, in _batch_scheduler
    self.batch_sizes[sched] = self._detect_batch_size(n_reordered_requests, pos)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1025, in _detect_batch_size
    batch_size = forward_batch()
                 ^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/accelerate/utils/memory.py", line 180, in decorator
    return function(batch_size, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1017, in forward_batch
    self._model_call(test_batch, **call_kwargs),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1154, in _model_call
    return self.model(inps).logits
           ^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 911, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_assistant/modeling_gemma4_assistant.py", line 167, in forward
    raise ValueError("inputs_embeds and shared_kv_states cannot be None.")
ValueError: inputs_embeds and shared_kv_states cannot be None.

Running loglikelihood requests:   0%|          | 0/100012 [00:01<?, ?it/s]

Auto-generated by error_analysis pipeline. cc @lvkaokao