Scoring appears to have stalled since Oct 23 19:50-20:37 KST (about 2 days) in Seasons 2, 3 and 4 (submissions accepted, no scores returned)

#10
by 1Dmeds - opened

Hello,

We are reporting what looks like a backend issue, in case it is not yet known.

Observed from three public leaderboards as of Oct 25 23:40 KST:

Season 2 (7,077 rows): most recent scored_at is Oct 23 20:37. Zero entries scored
on Oct 24 or Oct 25, across all teams. Daily scored counts: Oct 21 = 34, Oct 22 = 119, Oct 23 = 5, Oct 24 = 0, Oct 25 = 0.

Season 3 (5,316 rows): most recent scored_at is Oct 23 20:08. Zero entries scored
on Oct 24 or Oct 25. Daily scored counts: Oct 21 = 42, Oct 22 = 83, Oct 23 = 64, Oct 24 = 0, Oct 25 = 0.

Season 4 (4,270 rows): most recent scored_at is Oct 23 19:50. Zero entries scored
on Oct 24 or Oct 25. Daily scored counts: Oct 21 = 85, Oct 22 = 77, Oct 23 = 51, Oct 24 = 0, Oct 25 = 0.
Eight submissions we made on Oct 23 at 21:06 were accepted (HTTP 200) and have no score.
On Oct 26 at 16:13-16:17 we submitted 30 more; all were accepted and none is scored yet.

All three seasons stop within about 45 minutes of each other and none has scored anything
in the roughly two days since, so this does not look season-specific.

  • The submission endpoint returned HTTP 503 from roughly Oct 24 00:34 to 15:09 KST
    (~14.5 hours), then recovered. Submissions are accepted normally since then.
  • The acceptance responses return queue_position values that keep growing (57 through 86
    earlier; 118 through 147 for our Season 4 submissions on Oct 26). That is a direct signal
    from your own API that a backlog has built up and is not draining.

This suggests the submission API and the scoring worker recovered separately, and the
scoring worker may still be down.

Two questions:

  1. Is the scoring backlog expected to clear on its own, or does the worker need a restart?
  2. Submissions accepted during this window still consume a daily slot. Will they be scored
    once the worker resumes, or should we treat them as lost?

We are not asking for quota compensation and we are not disputing any score. We only need to
know whether submissions made now will eventually return results, so we can decide whether to
keep spending daily slots blind.

Thank you,
1Dmeds

FINAL_Bench org

채점은 λ©ˆμΆ”μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€. 멈좘 것은 μˆœμœ„ν‘œμž…λ‹ˆλ‹€. 그리고 보내주신 증거 ν•˜λ‚˜κ°€ λ°”λ‘œ κ·Έ κ³ μž₯의 μ¦μƒμž…λ‹ˆλ‹€.

보고가 μ •λ°€ν–ˆμŠ΅λ‹ˆλ‹€. 특히 queue_position 이 계속 μ»€μ§„λ‹€λŠ” 관찰이 원인을 νŠΉμ •ν•˜λŠ” 데 κ²°μ •μ μ΄μ—ˆμŠ΅λ‹ˆλ‹€.

μ‹€μΈ‘ (2026-10-26 16:48 UTC)

μ‹œμ¦Œ 제좜 κ²°κ³Ό 기둝됨 μ‹€μ œ 미채점 μˆœμœ„ν‘œμ— 올라온 수
μ‹œμ¦Œ2 7,219 7,219 0 7,058
μ‹œμ¦Œ3 5,359 5,359 0 5,308
μ‹œμ¦Œ4 4,444 4,428 16 (처리 쀑) 4,267

λ§ˆμ§€λ§‰ 채점은 10μ›” 26일 16:35 UTC μž…λ‹ˆλ‹€. 24일과 25일에도 채점이 λŒμ•˜μŠ΅λ‹ˆλ‹€. μˆœμœ„ν‘œμ— 올라온 μˆ˜μ™€ κ²°κ³Ό 기둝 수의 차이가 μ§€κΈˆ 보이지 μ•ŠλŠ” λΆ„λŸ‰μž…λ‹ˆλ‹€.

queue_position 에 λŒ€ν•˜μ—¬

/api/queue 의 scored λŠ” κ²°κ³Ό 파일 μˆ˜κ°€ μ•„λ‹ˆλΌ μˆœμœ„ν‘œ ν•­λͺ© μˆ˜μ—μ„œ μ˜΅λ‹ˆλ‹€. queued = total - scored 둜 κ³„μ‚°ν•©λ‹ˆλ‹€. μˆœμœ„ν‘œκ°€ 10μ›” 23일에 λ©ˆμΆ”λ©΄μ„œ scored κ°€ 고정됐고, 제좜이 λ“€μ–΄μ˜¬μˆ˜λ‘ queued 만 μ»€μ‘ŒμŠ΅λ‹ˆλ‹€. μ§€κΈˆ 값은 161, 51, 177 μž…λ‹ˆλ‹€. 저희 API κ°€ 적체가 μŒ“μ΄λŠ” κ²ƒμ²˜λŸΌ λ³΄μ΄λŠ” μ‹ ν˜Έλ₯Ό λ³΄λƒˆκ³ , κ·Έ 해석은 ν•©λ‹Ήν–ˆμŠ΅λ‹ˆλ‹€. μ‹€μ œ μ μ²΄λŠ” 0, 0, 16 μž…λ‹ˆλ‹€.

원인

μˆœμœ„ν‘œ 파일이 10MBλ₯Ό λ„˜μž μ €μž₯μ†Œκ°€ κ·Έ νŒŒμΌμ„ λŒ€μš©λŸ‰ μΆ”μ μœΌλ‘œ μžλ™ μ „ν™˜ν–ˆμŠ΅λ‹ˆλ‹€. λŒ€μš©λŸ‰ 파일 μ“°κΈ°κ°€ 쑰직 μ €μž₯μš©λŸ‰ ν•œλ„μ— κ±Έλ € 거절되고 μžˆμŠ΅λ‹ˆλ‹€. μž‘μ€ κ²°κ³Ό νŒŒμΌμ€ 영ν–₯을 λ°›μ§€ μ•Šμ•„ μ •μƒμœΌλ‘œ μ¨μ§‘λ‹ˆλ‹€. κ·Έλž˜μ„œ 채점은 돌고 곡개면만 멈좰 μžˆμŠ΅λ‹ˆλ‹€. μ „ν™˜ κ·œμΉ™μ΄ κ²½λ‘œκ°€ μ•„λ‹ˆλΌ 파일 μ΄λ¦„μœΌλ‘œ λ§žμ•„ μ„Έ μ‹œμ¦Œμ΄ 같은 λ‚  ν•¨κ»˜ λ§‰ν˜”μŠ΅λ‹ˆλ‹€.

μ§ˆλ¬Έμ— λŒ€ν•œ λ‹΅

  1. 적체가 슀슀둜 ν’€λ¦¬λŠ”μ§€, μ›Œμ»€ μž¬μ‹œμž‘μ΄ ν•„μš”ν•œμ§€: μž¬μ‹œμž‘μ€ ν•„μš” μ—†μŠ΅λ‹ˆλ‹€. μ›Œμ»€ 7κ°œκ°€ 돌고 μžˆμŠ΅λ‹ˆλ‹€. 16:44 UTC 에 18건이던 미채점이 16:48 에 16건으둜 μ€„μ—ˆμŠ΅λ‹ˆλ‹€. ν’€μ–΄μ•Ό ν•  적체 μžμ²΄κ°€ μ—†μŠ΅λ‹ˆλ‹€. 볡ꡬ가 ν•„μš”ν•œ 것은 μˆœμœ„ν‘œμž…λ‹ˆλ‹€.
  2. 이 κΈ°κ°„ 제좜이 μ±„μ λ˜λŠ”μ§€, 버렀진 κ²ƒμœΌλ‘œ 봐야 ν•˜λŠ”μ§€: 이미 μ±„μ λμŠ΅λ‹ˆλ‹€. 10μ›” 23일 21μ‹œ μ œμΆœλΆ„ 8건과 10μ›” 26일 μ œμΆœλΆ„ 30건을 ν™•μΈν–ˆκ³ , 1Dmeds λ‹˜ 계정에 미채점은 0κ±΄μž…λ‹ˆλ‹€. μ§€κΈˆ 남은 미채점 16건은 10μ›” 26일 15μ‹œ 30λΆ„λŒ€μ— λ“€μ–΄μ˜¨ λ‹€λ₯Έ ν•œ 묢음이고 μˆœμ„œλŒ€λ‘œ 처리되고 μžˆμŠ΅λ‹ˆλ‹€. 버렀진 것은 μ—†μŠ΅λ‹ˆλ‹€.

κ·ΈλŸ¬λ‹ˆ ν•˜λ£¨ λͺ«μ„ μ“°λ©΄μ„œ λˆˆμ„ 가리고 μ œμΆœν•˜λŠ” 상황은 μ•„λ‹™λ‹ˆλ‹€. κ²°κ³ΌλŠ” 남아 있고, 보이지 μ•ŠλŠ” κ²ƒλΏμž…λ‹ˆλ‹€.

503 ꡬ간(10μ›” 24일 00:34μ—μ„œ 15:09 KST)은 λ³„κ°œλ‘œ νŽ˜μ΄μ§€ μͺ½ λ¬Έμ œμ˜€κ³  λ³΅κ΅¬λμŠ΅λ‹ˆλ‹€. 제좜 κ²½λ‘œμ™€ 채점 κ²½λ‘œκ°€ λ”°λ‘œ λ³΅κ΅¬λλ‹€λŠ” 지적이 λ§žμŠ΅λ‹ˆλ‹€.

λ‹€μŒ

μˆœμœ„ν‘œ 볡ꡬ본은 μ„Έ μ‹œμ¦Œ λͺ¨λ‘ λ§Œλ“€μ–΄ 검증을 λ§ˆμ³€μŠ΅λ‹ˆλ‹€. 이미 μ˜¬λΌμ™€ μžˆλŠ” ν•­λͺ©μ„ κΈ€μž λ‹¨μœ„λ‘œ μž¬ν˜„ν•  수 μžˆλŠ”μ§€ λ¨Όμ € ν™•μΈν•˜κ³ , ν•˜λ‚˜λΌλ„ μ‚¬λΌμ§€κ±°λ‚˜ 값이 달라지면 μ˜¬λ¦¬μ§€ μ•Šλ„λ‘ ν•΄ λ‘μ—ˆμŠ΅λ‹ˆλ‹€. μ €μž₯μš©λŸ‰ 건이 ν•΄μ œλ˜λ©΄ λ°”λ‘œ μ μš©ν•˜κ³  이 글에 μ κ² μŠ΅λ‹ˆλ‹€.

μˆœμœ„ν‘œκ°€ κ²°κ³Όλ₯Ό λ”°λΌμ˜€κ³  μžˆλŠ”μ§€ λ³΄λŠ” 점검도 κ°μ‹œμ— λ„£κ² μŠ΅λ‹ˆλ‹€. 이번 κ³ μž₯은 저희 κ°μ‹œλ‘œλŠ” 보이지 μ•Šμ•˜κ³ , 두 번 λͺ¨λ‘ μ°Έκ°€μžκ°€ λ¨Όμ € μ•Œλ € μ£Όμ…¨μŠ΅λ‹ˆλ‹€.


Scoring never stopped. The leaderboard did. One piece of your own evidence is a symptom of that failure.

Your report was precise, and the observation that queue_position keeps growing is what pinned down the cause.

Measured at 2026-10-26 16:48 UTC

Season 2: 7,219 submitted, 7,219 results written, 0 unscored, 7,058 on the leaderboard.
Season 3: 5,359 submitted, 5,359 results written, 0 unscored, 5,308 on the leaderboard.
Season 4: 4,444 submitted, 4,428 results written, 16 in progress, 4,267 on the leaderboard.

The most recent score was written on 26 October at 16:35 UTC. Scoring also ran on the 24th and the 25th. The gap between results written and rows on the leaderboard is what is currently invisible.

About queue_position

In /api/queue, scored is taken from the number of leaderboard rows, not the number of result files, and queued is total - scored. When the leaderboard froze on 23 October, scored froze with it, so queued grew with every new submission. It currently reads 161, 51 and 177. Our own API produced a signal that looked like a growing backlog, and reading it that way was reasonable. The real backlog is 0, 0 and 16.

Cause

Once the rollup file passed 10MB, the repository moved it to large file tracking, and large file writes are being refused against an organization storage limit. Small result files are unaffected and keep writing, so scoring runs while the public view stays frozen. The tracking rule matches by file name rather than path, so all three seasons were blocked on the same day.

Your questions

  1. No restart is needed. Seven workers are running, and unscored submissions went from 18 at 16:44 UTC to 16 at 16:48. There is no backlog to drain. What needs recovery is the rollup.
  2. They are already scored. We checked the eight submissions from 23 October and the thirty from 26 October, and your account has zero unscored entries. The 16 that remain belong to a separate batch submitted around 15:30 UTC on 26 October and are being worked through in order. Nothing is lost.

So you are not spending daily slots blind. The results exist, they are just not visible yet.

The 503 window on 24 October was a separate problem on the page side and has recovered. Your inference that the submission path and the scoring path recovered separately is correct.

Next

Restored rollups are built and verified for all three seasons. We first confirmed that existing rows reproduce character for character, and the upload refuses to proceed if any existing row would disappear or change. It will be applied as soon as the storage item is cleared, and we will post here when it is.

We will also add a check that the rollup is keeping up with the results. Our monitoring could not see this failure, and on both occasions a participant found it first.

FINAL_Bench org

μ μš©λμŠ΅λ‹ˆλ‹€. μˆœμœ„ν‘œκ°€ λ³΅κ΅¬λμŠ΅λ‹ˆλ‹€.

μ‹œμ¦Œ μ˜¬λΌμ˜€μ§€ λͺ»ν–ˆλ˜ 채점 볡ꡬ ν›„ μˆœμœ„ν‘œ μ°Έκ°€μž ν™”λ©΄ λŒ€κΈ°
μ‹œμ¦Œ2 161 7,219 0
μ‹œμ¦Œ3 51 5,359 0
μ‹œμ¦Œ4 177 4,444 0

λ§μ”€ν•˜μ‹  queue_position 도 μ„Έ μ‹œμ¦Œ λͺ¨λ‘ 0으둜 λ‚΄λ €κ°”μŠ΅λ‹ˆλ‹€. κ·Έ 값이 μ»€μ§€λ˜ 것은 적체가 μ•„λ‹ˆλΌ μˆœμœ„ν‘œκ°€ 멈좘 κ²°κ³Όμ˜€κ³ , 지적해 μ£Όμ‹  덕뢄에 κ·Έ 지점을 μ°Ύμ•˜μŠ΅λ‹ˆλ‹€.

μˆœμœ„ν‘œλ₯Ό 10MB μ•„λž˜ 쑰각으둜 λ‚˜λˆ  μ“°λŠ” λ°©μ‹μœΌλ‘œ λ°”κΏ¨κ³ , 채점기도 같은 λ°©μ‹μœΌλ‘œ κ΅μ²΄ν–ˆμŠ΅λ‹ˆλ‹€. 올린 λ’€ λ˜μ½μ–΄ 원본과 κΈ€μž λ‹¨μœ„λ‘œ 같은지 ν™•μΈν–ˆμŠ΅λ‹ˆλ‹€.

μ μˆ˜λŠ” 이미 계산돼 있던 값이라 λ°”λ€Œμ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€. μž¬μ œμΆœμ€ ν•„μš” μ—†μŠ΅λ‹ˆλ‹€.


Applied. The leaderboard is restored.

Season 2: 161 entries now visible, 7,219 total. Season 3: 51, now 5,359. Season 4: 177, now 4,444. queued is now 0 for all three.

The growing queue_position you reported was not a backlog, it was the frozen rollup showing through, and your report is what located it.

The rollup is now written as text shards under the size limit, and the scorer was switched to the same format. We read the uploaded result back and confirmed it matches the source character for character. The scores themselves were already computed and are unchanged, so no resubmission is needed.

FINAL_Bench org

마감 μ‘°μ • κ²°κ³Όμž…λ‹ˆλ‹€. μ‹œμ¦Œ2 λ§ˆκ°μ€ 10μ›” 31일 κ·ΈλŒ€λ‘œμ΄κ³ , μ—°μž₯ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€.

보상을 μš”κ΅¬ν•˜μ§€ μ•ŠμœΌμ‹ λ‹€κ³  λ¨Όμ € λ°ν˜€ μ£Όμ…¨μ§€λ§Œ, μ•žμ„œ 별도 곡지λ₯Ό μ•½μ†ν–ˆμœΌλ―€λ‘œ κ²°κ³Όλ₯Ό μ „ν•©λ‹ˆλ‹€.

νŒλ‹¨ κ·Όκ±°λŠ” μ μˆ˜κ°€ 사라진 적이 μ—†κ³  μ§€κΈˆμ€ μ „λΆ€ λ³΄μΈλ‹€λŠ” μ μž…λ‹ˆλ‹€. ν˜„μž¬ μ „ μ‹œμ¦Œ 미채점 0, queued 0μž…λ‹ˆλ‹€.

남은 기간을 μœ„ν•΄ μ‹œμ¦Œ2 채점 λŠ₯λ ₯을 두 배둜 올렸고, μˆœμœ„ν‘œκ°€ κ²°κ³Όλ₯Ό λ”°λΌμ˜€λŠ”μ§€ λ³΄λŠ” 점검을 κ°μ‹œμ— λ„£μ—ˆμŠ΅λ‹ˆλ‹€. μ§€μ ν•˜μ‹  γ€Œλ§ˆκ° 직전에 λŒ€κΈ°μ—΄μ΄ κΈΈμ–΄μ Έ 같은 일이 반볡될 μ†Œμ§€γ€κ°€ λ°”λ‘œ κ·Έ 쑰치의 μ΄μœ μž…λ‹ˆλ‹€. λ‹€λ§Œ μ°Έκ°€μž 전원이 λ§ˆμ§€λ§‰ 날에 ν•œλ„λ₯Ό λ‹€ μ“°λ©΄ κ·Έλž˜λ„ λ°€λ¦½λ‹ˆλ‹€.

전체 κ³΅μ§€λŠ” λ³„λ„λ‘œ μ˜¬λ ΈμŠ΅λ‹ˆλ‹€.


The deadline decision: Season 2 closes on 31 October as scheduled, and will not be extended.

You said up front that you were not asking for quota compensation, but we promised a separate announcement, so here is the outcome.

The reasoning is that no score was lost and everything is visible now. Unscored submissions are at zero across all seasons and queued reads zero.

For the remaining days, Season 2 scoring capacity is doubled and monitoring now checks whether the rollup is keeping up with the results. Your point about the queue growing near a deadline is exactly why. If every entrant uses a full daily quota on the last day there will still be a wait.

A separate announcement covers this for all participants.

FINAL_Bench org

결정이 λ°”λ€Œμ—ˆμŠ΅λ‹ˆλ‹€. μ‹œμ¦Œ2 λ§ˆκ°μ„ 11μ›” 7일둜 1μ£Ό μ—°μž₯ν•©λ‹ˆλ‹€.

같은 λ‚  μ•žμ„œ γ€Œμ—°μž₯ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€γ€λΌκ³  μ μ—ˆμŠ΅λ‹ˆλ‹€. κ·Έ λ’€ 남은 κΈ°κ°„μ˜ 처리 λŠ₯λ ₯을 μ‹€μΈ‘ν–ˆκ³ , 결과에 따라 λ°”κΏ¨μŠ΅λ‹ˆλ‹€.

남은 4일에 λ“€μ–΄μ˜¬ 수 μžˆλŠ” μ΅œλŒ€ μ œμΆœλŸ‰μ΄ 720건(ν™œλ™ 계정 6κ³³ Γ— ν•˜λ£¨ 30건)인데 λ‹Ήμ‹œ λŠ₯λ ₯은 560κ±΄μ΄μ—ˆμŠ΅λ‹ˆλ‹€. λŠ₯λ ₯을 두 배둜 μ˜¬λ Έμ§€λ§Œ μ‘°μš©ν–ˆλ˜ 계정이 마감 전에 λŒμ•„μ˜€λ©΄ μ—¬μœ κ°€ μˆ˜μ‹­ 건으둜 μ€„μ–΄λ“­λ‹ˆλ‹€. μ§€μ ν•˜μ‹  γ€Œλ§ˆκ° 직전에 λŒ€κΈ°μ—΄μ΄ κΈΈμ–΄μ Έ 같은 일이 반볡될 μ†Œμ§€γ€κ°€ λ°”λ‘œ κ·Έ κ³„μ‚°μ—μ„œ ν™•μΈλμŠ΅λ‹ˆλ‹€.

보상을 μš”κ΅¬ν•˜μ§€ μ•ŠμœΌμ‹ λ‹€κ³  λ¨Όμ € λ°ν˜€ μ£Όμ…¨μ§€λ§Œ, 결정이 λ°”λ€Œμ—ˆμœΌλ‹ˆ μ „ν•΄ λ“œλ¦½λ‹ˆλ‹€. μƒˆ λ§ˆκ°μ€ 11μ›” 7일이고 μ ‘μˆ˜ νŽ˜μ΄μ§€μ™€ λ³΄λ“œ ν‘œκΈ°λ„ λ°”κΏ¨μŠ΅λ‹ˆλ‹€. 전체 κ³΅μ§€λŠ” #11 에 μˆ˜μ •ν•΄ μ˜¬λ ΈμŠ΅λ‹ˆλ‹€.


The decision has changed. Season 2 is extended by one week, to 7 November.

Earlier the same day we wrote that the deadline would not be extended. We then measured the scoring capacity available for the remaining days and changed the decision.

Up to 720 submissions could arrive in four days (6 recently active accounts at 30 per day) against a capacity of about 560. We doubled capacity, but if quiet accounts return before the deadline the margin falls to a few dozen entries. Your point about the queue growing near a deadline is exactly what that calculation confirmed.

You said up front that you were not asking for compensation, but the decision has changed so we are telling you. The new deadline is 7 November, and the submission page and board now show it. The full notice is revised in #11.

Sign up or log in to comment