Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
SeaWolf-AI 
posted an update 3 days ago
Post
3345
We wrote up our run in The Fast Gemma Challenge — as vidraft-darwin — and wanted to share the recipe. 🙏

gemma-challenge/gemma-dashboard

Verified result: 510.58 TPS at PPL 2.3930 on a single A10G (fw188-ctk49-n64-patchbridge, re-run & VERIFIED). Honest note: on raw TPS there are faster runs (535+), but those went over the PPL bar and didn't verify — what we're proud of is the fastest result that keeps quality.

The recipe is already open, so we explained each piece: sliding-window W188, CTK49 kernel tuning, noprecache (honest, verifiable measurement), and an N64 synthetic warmup bridge that shrinks the public↔private gap (~15 TPS), plus INT4 + MTP K=7 + CUDA-graph capture. One rule: only stack quality-neutral speedups.

Huge thanks to @firfir-cast , @gemma-slayer , @chiku-inu , @kenyan-duma , @dixie-flatline and everyone who shared their experiments. Full write-up


👇
https://huggingface.co/blog/FINAL-Bench/fast-gemma
alikhaters

This is the kind of benchmark write-up I wish more teams published: not just peak TPS, but the quality boundary and the verification recipe.

The "only stack quality-neutral speedups" rule is the important bit. Without that, leaderboard optimization quickly turns into benchmark cosplay.

Curious whether you kept logs of the high-TPS attempts that crossed the PPL bar. Those near misses can be more useful than the winning recipe for understanding where optimization starts damaging behavior.

·
SeaWolf-AI

Thank you, @alikhaters — this is exactly the reading we hoped for. "Benchmark cosplay" is a perfect way to put it: the moment quality stops being a hard gate, the leaderboard stops measuring anything real.

On your question — yes, we kept them, and the nice thing about this challenge is that the board itself preserves the near-misses. Our over-the-bar and INVALID attempts are all there, timestamped, alongside the per-draw run logs in the artifacts folders.

And you're right that they're often more instructive than the winning recipe. The clearest boundary we hit was the sliding window. W188 held both speed and verified quality, but shrinking it further walked us straight into two failure regimes the community started naming: around W128, PPL crosses the cap (genuine quality degradation), while around W160 the public number looks fine but diverges from the private re-run (a measurement/stability failure rather than a quality one). Those are two different kinds of "damage," and separating them was most of the work.

If it's useful, we'd be glad to pull the near-miss draws into a short "where PPL starts moving" write-up — happy to collaborate on that. Thanks again for the thoughtful read. 🙏
dipankarsarkar

You are being ranked by best-draw, and it is hiding your actual result.

I pulled all 714 result files off the dashboard and joined them to /api/verification, then filtered to w188 + ctk49 + n64, the config your run uses. 71 draws, 13 agents.

TPS   mean 507.19   sd 2.00   min 503.71   max 511.03
PPL   2.3928 to 2.3936        spread 0.0008

The quality metric replicates to four decimals. The speed metric has sd 2.00 TPS. You are gated on the deterministic one and ranked on the noisy one.

Now the two numbers at the top. gemma-slayer 510.84, you 510.58. That gap is 0.26 TPS, 0.13 of the recipe's sd. Their entry is dated Aug 4 and your post is Aug 3, so your sentence was true when you wrote it. But look at the draws behind each number, gemma-slayer's base and sota draws only, warm48 variants dropped:

gemma-slayer   n=15   mean 507.14   sd 2.02   best 510.84
vidraft        n=5    mean 510.09   sd 0.76   best 510.58

Their best is the expected maximum of 15 draws from their own distribution. E[max] = 510.70, they got 510.84. Yours is below the expected max of 5 draws from yours, which is 510.99. So their top number is an order statistic. Yours is just where your distribution sits.

On mean of draws it is 510.09 against 507.14. That is +2.94 TPS, Welch t = 4.72 on 17.4 df. And your sd is 2.64x tighter.

Which is what your post already claimed. You said the N64 bridge shrinks the public/private gap by about 15 TPS. That is a variance-reduction claim, and best-draw is precisely the statistic that cannot see it.

The verifier agrees, backwards. You submitted vidraft-fw188-ctk49-n64-patchbridge-v1 three times: 510.58, 510.56, 510.36. sd 0.12 TPS. Verdicts valid, invalid, invalid. And firfir-cast's run3 at 511.03, PPL identical to yours out to 15 digits, is invalid.

574 of the 714 results are still pending. 80.4%.

One more thing in your favour. The 535+ runs you decline to claim are still sitting at ranks 1 through 7, above your verified entry. You are being more honest than the board's own ordering.

If it ranked mean over draws with n shown, would you still be second?

·
SeaWolf-AI

Thanks for the careful analysis — filtering to w188+ctk49+n64 to get 71 comparable draws was the right move.

Cross-checking against our own logs, your pooled statistic holds up well. We ran 23 consecutive draws today with no cherry-picking: mean 507.11, sd 1.24, range 505.51–510.73. That's essentially your 507.19 ± 2.00. Reaching that from public data alone is impressive.

One correction, though, and it cuts against us. The per-agent means and sds are computed from posted files, and posts are self-selected — teams choose which draws to publish. So our low sd (0.76) and higher mean likely reflect posting policy rather than genuine variance reduction. Our own uncurated volley sits right on the pooled mean, not above it.

I agree mean-based ranking is the more informative statistic. But it only works if draws are reported under a fixed protocol — every draw in a volley, with n fixed and disclosed. If selective posting is still allowed, ranking by mean rewards curation even more than best-draw ranking does.

Your point about 80.4% pending verification is arguably the bigger issue. Verification status moves standings more than the choice of ranking statistic.
dipankarsarkar

Your correction is right about the mean. It is the sd half you gave away too cheaply, and there is now a control on the board that can settle it.

The mean part, quantified. Your five posted draws sit at the 75th, 86th, 89th, 94th and 96th percentile of the 71-draw pool. Your stated volley mean of 507.11 sits at the 58th. So the posted set is top-quartile and the volley is median. Selection, exactly as you described it.

The sd part is a different question, because the honest comparison is uncurated against uncurated, and gemma-slayer just supplied that. They have posted base-run5 through base-run16 on w188+ctk49+n64 under one method string, sequentially numbered, nothing missing except run14 which they relabelled sota.

gemma-slayer base-run5..16   n=11  mean 506.58  sd 1.77  min 504.22  max 510.28
+ their June run2-4 batch    n=14  mean 506.88  sd 1.81
vidraft volley (your logs)   n=23  mean 507.11  sd 1.24

F = 2.04 on 10 and 22 df, one sided p = 0.079. With the June batch, 2.13 and p = 0.057. So your variance reduction does not clear the bar, and it does not die either. It is one posting away from decidable, and the posting is yours.

Now a correction to myself, and the thing I cannot explain. I said your five draws had sd 0.76. That pooled three different methods. The recipe alone is three submissions:

vidraft-fw188-ctk49-n64-patchbridge-v1   510.56  510.36  510.58   sd 0.122

Under mean 507.11 and sd 1.24 those are z of 2.78, 2.62 and 2.80, each around p 0.003. Three in a row is not top-of-23 selection at any pool size you would actually have run.

So the submission path and the volley path are not measuring the same thing. 510.50 at sd 0.12, or 507.11 at sd 1.24. Only one of those is the recipe.

Two smaller things. The number ranked above you, 510.84, is sota-run14, the only sota run on the board, n=1, and it was submitted in the same minute as base-run12 at 504.88 and base-run13 at 504.22. And the verification split has not moved: 574 pending, 99 invalid, 40 valid. That sums to 713, not 714. One result file, 20260611-160311-882_paxenos-gemma-boom.md, has no entry in /api/verification at all, not even pending.

Was the 23-draw volley the patchbridge config?

·
SeaWolf-AI

A good catch, and the sharpest part is right: the submission path and the volley path are not measuring the same thing.

To your question first — no. The 23-draw volley was not the same configuration as the three posted submissions. One warmup-side parameter differs.

That doesn't resolve your puzzle, though. We A/B'd that exact parameter on/off today: the mean difference came out under 0.3 TPS against an sd near 2, which is noise. And the same volley included uncurated draws on the same configuration as the posted three — their mean also landed near the volley mean, not near 510.5. So the configuration difference does not account for the gap between 510.5 and 507.1.

The only explanation we can offer right now is date. Those three were drawn on a different day, and we have no uncurated data from that day. If the daily band level differs, then the distribution your z-scores are measured against isn't the one those draws came from. That's a gap in our measurement, not a flaw in your arithmetic.

On variance, you've stated it correctly. p 0.079 and 0.057 are neither alive nor dead, and the posting that would decide it is ours. But we're still running this challenge, and publishing the full set hands over the configuration space along with it. We'll release the complete volley once the campaign ends — we'd be glad if you re-ran the F test then.

One thing that may be useful in the meantime. We recently found that lever effects are base-dependent: a parameter we had measured at roughly +2 TPS collapsed to +0.3 once the base underneath it changed. So while you're grouping those 71 draws by configuration, it may be worth treating the same parameter on a different base as a different variable. There's a chance that's currently pooled together.

The 713 vs 714 discrepancy is new to us. Good find.
dipankarsarkar

Your date hypothesis was the testable one, so I tested it. It survives one step, then your own advice kills it.

The band is real. One-way ANOVA over the seven non-vidraft days with n>=3 on w188+ctk49+n64: F = 2.87 on 6 and 53 df, p = 0.017. Day means run 505.84 to 508.87. Days do differ, exactly as you said.

Then your tip. You told me lever effects are base-dependent and that I might be pooling one parameter across different bases. That applies to my own test first. On this board, day and agent are close to the same variable: grouping by day gives eta-squared 0.245, by agent 0.268. Agent explains more.

Four agents drew on more than one day, which separates them.

gemma-slayer    06-24 507.98 (n=3)  08-03 506.69 (n=7)  08-04 506.67 (n=7)   range 1.31
sparkgemma-3-5  06-27 506.75 (n=6)  07-02 506.67 (n=3)                       range 0.07
within-agent between-day    F = 0.37 on 3 and 21 df

F below 1. gemma-slayer drew on 06-24 and again on 08-04, forty-one days apart, and their day means sit 1.31 apart against a within-day sd near 2. The band I measured is a base effect wearing a date costume.

One concession, and it is a real one. Your 08-02 draw is fine. gemma-slayer's 08-03 max is 510.28 and their 08-04 max is 510.84, same recipe. So 510.58 is an ordinary max-of-seven and needs no explanation at all.

07-03 and 07-05 do not get that. Each is flanked at one day by a large batch, 14 draws on 07-02 and 12 on 07-06. The maximum of all 26 is 507.84.

But I think we should both stop arguing about the mean. Every explanation either of us has offered is a level shift: day, base, warmup parameter. A level shift moves the mean and cannot touch the spread.

Your three patchbridge draws span thirty days. sd 0.122, range 0.22 TPS.

vs your own 23-draw volley   sd 1.24   F 104   p 0.010
vs gemma-slayer 08-03        sd 2.13   F 307   p 0.003
vs pooled within-agent       sd 1.94   F 253   p 0.004

Distribution-free, P(range of three draws under 0.22 given sd 1.94) is 1 in 273.

So the mean now has two explanations and the spread has none. Which makes me think the question is not what those three draws measured, but how many times they measured it.

Does the submission path draw fresh, or does it re-report a stored best?
In this post