Season opening dates, Season 1 ties, and how resubmissions are scored

#1
by 1Dmeds - opened

Hello organizers, thank you for running the Open Quantum Challenge.

  1. Opening dates. The Space lists both seasons as opening on 2026-10-11, while the harness README lists 2026-10-20 for Season 1 and 2027-02-10 for Season 2. Which dates govern prize eligibility, and do entries submitted now count toward the final ranking?

  2. Season 1 ties. Several entries already have MAE = 0.0 using exact diagonalization, which the quickstart itself uses. Are exact-diagonalization entries eligible for prizes, and how are equal scores ranked (submission time, or another criterion)?

  3. Resubmissions. When a participant submits more than once on a board, does the leaderboard keep the best score or the latest one?

Thank you for clarifying.

FINAL_Bench org

Thanks for the clear questions. Answers below.

  1. Opening dates. Both seasons are open as of 2026-10-11, which is what the Space shows. The README has been corrected to match, so there is no longer a discrepancy. Every submission from 2026-10-11 onward counts toward the final rankings. Deadlines are 2027-02-28 (Season 1) and 2027-03-31 (Season 2).

  2. Season 1 ties and exact diagonalization. You are right. For the current small Season 1 systems, exact diagonalization is a valid solution, so an MAE of 0.0 is reachable and multiple entries will tie there. Season 1 is therefore provisional: we are revising it to add larger systems that are beyond exact reach and to rank by achieved energy, so that method quality separates entries rather than copying the exact answer. Season 2 (QEC) is unaffected and runs as is.

  3. Resubmissions. The leaderboard shows your best score, not the most recent, so entering multiple times is fine.

We will post the Season 1 revision in this thread when it lands. Thanks for helping tighten the rules.

FINAL_Bench org

Follow-up: Season 1 has been revised to remove the exact-diagonalization tie problem, and the changes are live now.

What changed:

  • Ranking is by the lowest total ground-state energy you achieve across all systems, not MAE versus the exact answer. By the variational principle any valid state gives an energy at or above the true ground state, so the task is to get as low as you can. A reported energy below the exact ground state is rejected as unphysical.
  • You must report every system. Incomplete submissions are not ranked.
  • Larger systems were added (2x4 and 3x3 Hubbard); the set now has 9 systems.
  • Ties are broken by submission time, earliest first.
  • The known floor, the sum of exact ground-state energies, is shown on the board as a reference anchor.

Because the system set and metric changed, the Season 1 board was reset. Please resubmit for the new 9-system set. Season 2 (QEC) is unchanged. Thanks again for catching this.

Thanks again for the quick answers and the Season 1 revision. Two short follow-ups on Season 2 (QEC):

  1. Is the final Season 2 ranking based on the current 10,000-shot public set, or will submissions be re-scored on a separate private/fresh set at the deadline? If re-scored, will it use our submitted predictions or our decoder code?
  2. How are ties broken in Season 2 (e.g., earliest submission time, as in Season 1)?
    Thank you!

One more Season 1 note: the current #1 entry reports a total of -36.44490, which is below the exact minimum floor of -36.44480 shown on the board. The rules say energies below the exact ground state are rejected as unphysical. Could you check whether a tolerance is being applied, and clarify how sub-floor entries are handled? Thanks!

FINAL_Bench org

On the two open follow-ups.

Season 2 (QEC) ranking: the final ranking uses the published detector set (qec/detectors.npy + dem.txt), scored against the privately held ground truth. There is no separate private-set re-score; scoring is on your submitted predictions (decoded shots), not on decoder code. Ties are broken by submission time, earliest first, consistent with Season 1.

Season 1 sub-floor entry: the displayed floor (about -36.4448) is the sum of the exact per-system ground-state energies. The physical-validity gate is applied per system with a numerical tolerance (0.02 Ha) to allow for solver rounding, so a total that sits about 0.0001 below the rounded floor passes as numerical noise rather than a true violation of the variational bound. We will tighten the display so a within-tolerance total is not shown below the floor. Thank you for flagging it.

Hi SeaWolf-AI, I noticed that on both boards (Quantum Simulation and QEC Decoder), my entry (1Dmeds) is now listed under "anchors" instead of the ranked entries, and the ranked list no longer includes it. Was this intentional? If it is a display issue, could you restore it? If it is intentional, could you explain on what basis an entry becomes an anchor and whether anchors are eligible for the prizes? Thank you.

FINAL_Bench org

Hi 1Dmeds - thanks for flagging this. Your entries are ranked, not anchors; that earlier state was a classification error on our side and it is now corrected. You are currently #1 on both the Quantum Simulation and QEC Decoder boards and remain fully eligible for the prizes. Anchors are reference baselines only (the exact-minimum floor and reference bounds) and never compete. Nothing needs changing on your end.

Sign up or log in to comment