Submission failed during inference twice (team TraVELers) โ€” #6 workaround didn't help

#12
by andrew0111 - opened

Hi @grossanchez ,

Our team (TraVELers) has submitted twice and both attempts ended with the same message: "Submission failed during inference. Verify that the artifact entrypoint runs successfully and that its CUDA/PyTorch stack supports the official NVIDIA A100 (sm_80) evaluation GPU."

  • Oct 24, 11:40 PM
  • Oct 25, 12:15 AM โ€” a resubmission with identical layers (only the notes line changed, following the workaround in #6)

Before resubmitting we checked the artifact locally with the devkit:

  • torch.cuda.get_arch_list() (torch 2.8.0+cu128) includes sm_80, and the flash-attn 2.8.3 wheel contains sm_80 kernels;
  • the full self-eval (self_eval.py --artifact โ€ฆ --gpus โ€ฆ, reconstructed from the uploaded layers with --network none) completes on the val split and passes validate_submission.py;
  • the entrypoint also runs as a non-root user on a read-only root filesystem.

Since the resubmission with identical layers (the #6 workaround) failed the same way, it looks like something in the evaluation environment that we can't reproduce locally. Could you share the actual error from the evaluation job so we can fix it, and could these two attempts be refunded as in #6?

Thanks!

Update: our third submission went through. The only change was that we stopped installing packages with apt in the Dockerfile. Our apt layer (for ffmpeg) included /etc/ld.so.cache and other system files, which get applied over the evaluation machine's / and may hide the GPU driver libs. We now install ffmpeg from the imageio-ffmpeg pip wheel (it bundles a static ffmpeg inside the venv) and keep everything under /opt. Posting in case it helps others see the same error.

Sign up or log in to comment