Commit History
-
Let callers set the critic's request parameters
VLMQualityJudge built its chat-completions request with fixed parameters, so the reasoning and sampling settings used for the critic could not be configured: callers reproducing a specific evaluation setup had no way to pin e.g. reasoning_effort. Add an extra_body passthrough (default None, so the request is unchanged unless a caller sets it) and raise the critic's max_tokens default to 16384. With reasoning models the thinking tokens come out of the same budget (measured up to ~6.7k tokens for gemini-3.1-pro at reasoning_effort=high), which leaves little headroom under 8192 for the analysis JSON itself.9ae57c4 -
Draw one seed base per item when seed_base is unset
With RunnerConfig.seed_base=None the generator was called with seed=None, so every iteration re-rolled the noise. The best-of-N loop then compared images that differed by both the rewritten prompt and the noise, so a score change could not be attributed to the rewrite, and a run could not be reproduced. Draw a single base per item and reuse it across iterations (sample slot j always gets base + j), which is how an explicit seed_base already behaves. The drawn value is recorded on the result as "seed_base" so a run can be reproduced afterwards.9d3c345 -
Compare per-iteration seeds as a set in the parallel-generation test
Samples within an iteration are generated concurrently, so the order in which the fake generator records their seeds is not deterministic. The test compared the recorded list directly and failed intermittently (about 1 run in 12 locally). Compare each iteration's seeds as a sorted group, which still asserts that the same seeds are reused across iterations.51ee80f
-
Super-squash branch 'main' using huggingface_hub
mingyuliutw +8: CYChenv, tangyue0820, fferroni, harrim-nv, liang1225, shilinzhu-nvidia, mli0603, mbalaNV Co-authored-by: CYChenv <CYChenv@users.noreply.huggingface.co> Co-authored-by: tangyue0820 <tangyue0820@users.noreply.huggingface.co> Co-authored-by: fferroni <fferroni@users.noreply.huggingface.co> Co-authored-by: harrim-nv <harrim-nv@users.noreply.huggingface.co> Co-authored-by: liang1225 <liang1225@users.noreply.huggingface.co> Co-authored-by: shilinzhu-nvidia <shilinzhu-nvidia@users.noreply.huggingface.co> Co-authored-by: mli0603 <mli0603@users.noreply.huggingface.co> Co-authored-by: mbalaNV <mbalaNV@users.noreply.huggingface.co>754f0cd