Spaces:
Paused
Tier 2d: make the AI bench measure what it claims to measure
Browse filesThe side-swap could not detect the thing it exists to detect. Its header
says "a side-symmetry bug would show up as those two disagreeing", but
duelSeed hashed the candidate names in ORDER while runSwappedDuel calls
runDuel(a,b) then runDuel(b,a) β so the two halves differed in seat AND
map, and any disagreement was confounded with ordinary seed variance.
Sorting the pair into the hash makes it a genuine paired comparison, which
is what swapAsym (the map-side control) already was. No result test needed
re-baselining: they compare two runs of the same call, not frozen values.
Swiss standings ranked raw win totals across unequal match counts. A bye
is scorable-neutral since e9ad1d0, but its recipient plays one fewer
pairing β 16 fewer chances to win under the CLI defaults β so sorting on
totals ranked "everyone who avoided a bye, then everyone who didn't": a
2W-2L candidate at 50% finished below a 3W-3L candidate at 50% purely for
having played more. That is the mirror image of the bug e9ad1d0 fixed.
rankStandings now sorts on win RATE with wins as the tie-break, and the
existing most-wins-first test is UPDATED to the new contract rather than
deleted.
And the round-1 bye was decided by CLI argument order: every standing is
0-0, the sort is stable, so the loop always took the last-LISTED
candidate. Tie-breaking the pairing order by name fixes it. That change
moved which candidate draws the bye in one existing fixture, whose winner
then never fought β retargeted by naming rather than by weakening the
assertion, with a comment on why the names matter.
tickSelfPlay's ordering claim was false and is now stated honestly. The
comment said both controllers think on the IDENTICAL pre-tick snapshot;
runAI(β¦, "player") is not a read, and measured over 400 think-cycles the
"ai" seat saw already-changed orders on 13% of them and already-founded
buildings on 1%. The channel that feeds decisions is `buildings`, so a
turret founded microseconds earlier is answerable a full think cycle
sooner β small, but fixed in direction, always favouring the same seat.
Making the claim TRUE means either a frozen pre-tick read or moving the
"ai" call out of sim.js's hardcoded pipeline; both restructure the tick
contract and belong in Tier 3. The comment now says exactly that, and a
characterisation test pins the bound to one frame and one direction so it
cannot widen unnoticed.
Speed: the Swiss combinatorics were being proved by simulating real
15-40 sim-minute matches. tools/ailab.js now exposes buildSwissBracket
with the match runner injected β the idiom pairRound's own test already
used and explained β and six schedule/bookkeeping tests run on a
deterministic stub. Full suite 2m17s -> 1m34s; the non-ailab suite is 22s.
The remaining ailab time is genuine AI-behaviour simulation (does the
skirmisher actually fight, does provocation fade) which I left alone:
shrinking those risks them no longer discriminating, which is the failure
mode this whole tier is about.
Suite: 1998 pass, 0 fail. typecheck clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZ2MLqDX3grEPTJpwivWZX
- test/ai-selfplay.test.js +29 -0
- test/ailab.test.js +98 -24
- tools/ailab.js +45 -6
- tools/selfplay.js +22 -5
|
@@ -25,6 +25,7 @@ import { createGameState, makeUnit, makeBuilding } from "../engine/state.js";
|
|
| 25 |
import { tick } from "../engine/sim.js";
|
| 26 |
import { runAI } from "../engine/ai.js";
|
| 27 |
import { accrueActionBudget, canAct, spend } from "../engine/aiCommon.js";
|
|
|
|
| 28 |
import { visibleThreatsNearHome, pickNextUnitType } from "../engine/aiMilitary.js";
|
| 29 |
import { updateFog, isVisibleAt } from "../engine/fog.js";
|
| 30 |
import { mulberry32 } from "../engine/rng.js";
|
|
@@ -296,3 +297,31 @@ test("two DIFFERENT seeds produce different self-play results (the fingerprint i
|
|
| 296 |
const { state: b } = runSelfPlayMatchTo(cfg(2), 60);
|
| 297 |
assert.notEqual(fingerprint(a), fingerprint(b), "two different seeds must not coincidentally fingerprint identically");
|
| 298 |
});
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
import { tick } from "../engine/sim.js";
|
| 26 |
import { runAI } from "../engine/ai.js";
|
| 27 |
import { accrueActionBudget, canAct, spend } from "../engine/aiCommon.js";
|
| 28 |
+
import { readFileSync } from "node:fs";
|
| 29 |
import { visibleThreatsNearHome, pickNextUnitType } from "../engine/aiMilitary.js";
|
| 30 |
import { updateFog, isVisibleAt } from "../engine/fog.js";
|
| 31 |
import { mulberry32 } from "../engine/rng.js";
|
|
|
|
| 297 |
const { state: b } = runSelfPlayMatchTo(cfg(2), 60);
|
| 298 |
assert.notEqual(fingerprint(a), fingerprint(b), "two different seeds must not coincidentally fingerprint identically");
|
| 299 |
});
|
| 300 |
+
|
| 301 |
+
test("the self-play think ordering stays bounded to one cycle and one direction (T2 characterisation)", () => {
|
| 302 |
+
// NOT an assertion that the ordering is fair β it isn't, and tools/selfplay.js now says so
|
| 303 |
+
// plainly. This pins the BOUND: the "player" controller is driven before tick()'s own runAI for
|
| 304 |
+
// "ai", so within a single frame the ai seat may read what the player seat just did β but never
|
| 305 |
+
// the reverse, and never across more than that one frame. If a future change widens the window
|
| 306 |
+
// (a second player-side pass, or state mutated between the two), this goes red.
|
| 307 |
+
const state = createSelfPlayState({ planetId: "korrath", seed: 5150 });
|
| 308 |
+
assert.ok(state.playerAi, "fixture sanity: a second controller exists");
|
| 309 |
+
|
| 310 |
+
// Both controllers share THINK_INTERVAL and start together, so they think on the same frames.
|
| 311 |
+
assert.equal(state.ai.think, state.playerAi.think,
|
| 312 |
+
"the two think countdowns start in lockstep β that is WHY the ordering matters");
|
| 313 |
+
|
| 314 |
+
const seen = [];
|
| 315 |
+
for (let i = 0; i < 40; i++) {
|
| 316 |
+
const before = state.buildings.size;
|
| 317 |
+
tickSelfPlay(state, 0.1);
|
| 318 |
+
seen.push(state.buildings.size - before);
|
| 319 |
+
}
|
| 320 |
+
assert.ok(seen.every(d => Number.isFinite(d)), "the loop actually ran");
|
| 321 |
+
// The one-frame bound expressed structurally: tickSelfPlay makes exactly ONE player-side think
|
| 322 |
+
// call per frame, ahead of exactly one ai-side call inside tick().
|
| 323 |
+
const src = readFileSync(new URL("../tools/selfplay.js", import.meta.url), "utf8");
|
| 324 |
+
const body = src.slice(src.indexOf("export function tickSelfPlay"), src.indexOf("\n}", src.indexOf("export function tickSelfPlay")));
|
| 325 |
+
assert.equal((body.match(/runAI\(/g) || []).length, 1,
|
| 326 |
+
"exactly one explicit runAI call in tickSelfPlay β a second would widen the information edge");
|
| 327 |
+
});
|
|
@@ -26,7 +26,7 @@ import { join } from "node:path";
|
|
| 26 |
import {
|
| 27 |
run, labWorld, summarise, score, applyOverrides, CHECKS, OPPONENTS, WORLDS, WEIGHTS, runLeaderboard,
|
| 28 |
runDuel, runRoundRobin, pinnedDuelDials, runSwappedDuel, runDuelBrackets, runRoundRobinSwapped, runSearch,
|
| 29 |
-
runSwissTournament, pairRound, snapshotTables, restoreTables,
|
| 30 |
} from "../tools/ailab.js";
|
| 31 |
import { STRATEGIES } from "../engine/aiStrategy.js";
|
| 32 |
import { DIFFICULTY_OPTIONS } from "../engine/aiDifficulty.js";
|
|
@@ -749,9 +749,29 @@ test("Swiss needs at least 2 candidates, and rejects duplicate names up front (r
|
|
| 749 |
assert.throws(() => runSwissTournament(dup, shortSwiss()), /duplicate candidate name/);
|
| 750 |
});
|
| 751 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 752 |
test("default rounds follow ceil(log2(n)), and every round pairs at most floor(n/2) matches", () => {
|
| 753 |
-
const candidates =
|
| 754 |
-
const
|
| 755 |
assert.equal(bracket.rounds, 3, "ceil(log2(8)) = 3, at least the floor of 3");
|
| 756 |
assert.equal(bracket.roundsLog.length, 3);
|
| 757 |
for (const { byeName, matches } of bracket.roundsLog) {
|
|
@@ -762,8 +782,8 @@ test("default rounds follow ceil(log2(n)), and every round pairs at most floor(n
|
|
| 762 |
|
| 763 |
test("Swiss runs strictly fewer pairings than full round-robin for a pool large enough for it to matter", () => {
|
| 764 |
const n = 10;
|
| 765 |
-
const candidates =
|
| 766 |
-
const
|
| 767 |
const swissPairings = bracket.roundsLog.reduce((a, r) => a + r.matches.length, 0);
|
| 768 |
const roundRobinPairings = (n * (n - 1)) / 2;
|
| 769 |
assert.ok(swissPairings < roundRobinPairings,
|
|
@@ -773,8 +793,8 @@ test("Swiss runs strictly fewer pairings than full round-robin for a pool large
|
|
| 773 |
test("byes rotate: nobody gets a second bye while another candidate hasn't had one yet", () => {
|
| 774 |
// 5 candidates (odd) over enough rounds that everyone gets exactly one bye before anyone gets a
|
| 775 |
// second β 5 rounds guarantees every candidate has been the odd one out exactly once.
|
| 776 |
-
const candidates =
|
| 777 |
-
const
|
| 778 |
const byes = bracket.roundsLog.map(r => r.byeName);
|
| 779 |
assert.equal(new Set(byes).size, 5, `every candidate should get exactly one bye across 5 rounds of 5, got: ${byes.join(", ")}`);
|
| 780 |
});
|
|
@@ -783,9 +803,9 @@ test("a bye is scorable-neutral: it adds to nobody's wins or losses, only its ow
|
|
| 783 |
// A bye is a candidate the schedule couldn't pair, not a match anyone won β crediting it as a
|
| 784 |
// win (at ANY size) lets a candidate that never fights outrank one that actually won something.
|
| 785 |
// The only thing a bye should move is `byes` (bookkeeping/reporting), never `wins`/`losses`.
|
| 786 |
-
|
| 787 |
-
const opts =
|
| 788 |
-
const
|
| 789 |
const round = bracket.roundsLog.find(r => r.byeName);
|
| 790 |
assert.ok(round, "fixture: an odd pool must produce a bye somewhere");
|
| 791 |
// byeMatchCount is purely informational β "how big a real pairing this round would have been" β
|
|
@@ -800,20 +820,22 @@ test("a bye never lets a candidate that structurally cannot fight tie or beat on
|
|
| 800 |
// neverInitiates + a zero standing-army cap (cannot ever field a fight in skirmish) draws a bye,
|
| 801 |
// and a genuine winner must still rank strictly above it β a bye must never be worth what a real
|
| 802 |
// win is worth.
|
| 803 |
-
|
| 804 |
-
|
| 805 |
-
|
| 806 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 807 |
});
|
| 808 |
-
|
| 809 |
-
// candidate draws the round's bye and therefore never fights anyone at all.
|
| 810 |
-
const candidates = [normal, crippled(1), crippled(2)];
|
| 811 |
const opts = shortSwiss({ worlds: ["korrath"], minutes: 15, seeds: 1, rounds: 1 });
|
| 812 |
const [bracket] = runSwissTournament(candidates, opts);
|
| 813 |
const byeRecipient = bracket.roundsLog[0].byeName;
|
| 814 |
assert.ok(byeRecipient, "fixture: 3 candidates is odd, a bye must fire");
|
| 815 |
const byeRow = bracket.standings.find(s => s.name === byeRecipient);
|
| 816 |
-
const normalRow = bracket.standings.find(s => s.name === "normal");
|
| 817 |
assert.equal(byeRow.wins, 0, "a candidate that never fought must have zero wins, bye or not");
|
| 818 |
assert.ok(normalRow.wins > byeRow.wins,
|
| 819 |
`the genuine winner (${normalRow.wins} wins) must rank strictly above a bye recipient that never fought (${byeRow.wins} wins)`);
|
|
@@ -827,9 +849,9 @@ test("rematch avoidance: with enough candidates relative to rounds, no pairing r
|
|
| 827 |
// with the old algorithm). pairRound now backtracks to find a zero-repeat matching whenever one
|
| 828 |
// exists, so this checks across several seed bases β including the ones independently confirmed
|
| 829 |
// to break the old algorithm β rather than trusting a single roll.
|
| 830 |
-
const candidates =
|
| 831 |
for (const seedBase of [1, 6, 7, 9, 10]) {
|
| 832 |
-
const
|
| 833 |
const seen = new Set();
|
| 834 |
for (const { matches } of bracket.roundsLog) {
|
| 835 |
for (const res of matches) {
|
|
@@ -883,12 +905,15 @@ test("Swiss standings: total wins equal total losses β a bye contributes to ne
|
|
| 883 |
assert.equal(totalWins, totalLosses, "every win must have a matching loss somewhere β a bye must add to neither");
|
| 884 |
});
|
| 885 |
|
| 886 |
-
test("standings are sorted
|
| 887 |
-
|
| 888 |
-
|
|
|
|
|
|
|
|
|
|
| 889 |
for (let i = 1; i < bracket.standings.length; i++) {
|
| 890 |
const prev = bracket.standings[i - 1], cur = bracket.standings[i];
|
| 891 |
-
assert.ok(prev
|
| 892 |
`standings order violated at index ${i}: ${JSON.stringify(prev)} before ${JSON.stringify(cur)}`);
|
| 893 |
}
|
| 894 |
});
|
|
@@ -976,3 +1001,52 @@ test("this suite leaves the shipped strategy table exactly as it found it (T2)",
|
|
| 976 |
assert.equal(STRATEGIES.aggressive.garrisonMult, 0.4, "the shipped garrisonMult, not a test's override");
|
| 977 |
assert.equal(STRATEGIES.aggressive.attackTimeoutMult, 0.55);
|
| 978 |
});
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
import {
|
| 27 |
run, labWorld, summarise, score, applyOverrides, CHECKS, OPPONENTS, WORLDS, WEIGHTS, runLeaderboard,
|
| 28 |
runDuel, runRoundRobin, pinnedDuelDials, runSwappedDuel, runDuelBrackets, runRoundRobinSwapped, runSearch,
|
| 29 |
+
runSwissTournament, pairRound, rankStandings, buildSwissBracket, snapshotTables, restoreTables,
|
| 30 |
} from "../tools/ailab.js";
|
| 31 |
import { STRATEGIES } from "../engine/aiStrategy.js";
|
| 32 |
import { DIFFICULTY_OPTIONS } from "../engine/aiDifficulty.js";
|
|
|
|
| 749 |
assert.throws(() => runSwissTournament(dup, shortSwiss()), /duplicate candidate name/);
|
| 750 |
});
|
| 751 |
|
| 752 |
+
/* ---- Swiss SCHEDULE tests, on synthetic results ---------------------------------------------
|
| 753 |
+
These four used to drive real 15-40 sim-minute matches to assert pure combinatorics β round
|
| 754 |
+
count, pairings per round, bye rotation, rematch avoidance β roughly 50 s of the suite's wall
|
| 755 |
+
clock spent proving arithmetic. tools/ailab.js now exposes buildSwissBracket with the match
|
| 756 |
+
runner injected, so they run on a stub in milliseconds. This is the idiom pairRound's own test
|
| 757 |
+
already used, and explained: "isolates the pairing algorithm from match simulation entirely."
|
| 758 |
+
The genuinely end-to-end tests (a strong candidate beating a crippled one, determinism,
|
| 759 |
+
override isolation) still run real matches. */
|
| 760 |
+
|
| 761 |
+
// A deterministic stand-in for runSwappedDuel: alphabetically-earlier name wins, so standings move
|
| 762 |
+
// in a predictable way without simulating anything.
|
| 763 |
+
const fakePair = (a, b) => ({
|
| 764 |
+
aName: a.name, bName: b.name, n: 2, draws: 0,
|
| 765 |
+
aWins: a.name < b.name ? 2 : 0,
|
| 766 |
+
bWins: a.name < b.name ? 0 : 2,
|
| 767 |
+
avgMargin: 0, rows: [],
|
| 768 |
+
});
|
| 769 |
+
const field = n => Array.from({ length: n }, (_, i) => ({ name: `C${i}`, strategy: "default" }));
|
| 770 |
+
const swissRounds = n => Math.max(3, Math.ceil(Math.log2(n)));
|
| 771 |
+
|
| 772 |
test("default rounds follow ceil(log2(n)), and every round pairs at most floor(n/2) matches", () => {
|
| 773 |
+
const candidates = field(8);
|
| 774 |
+
const bracket = buildSwissBracket(candidates, swissRounds(8), { worlds: ["korrath"], seeds: 1 }, fakePair);
|
| 775 |
assert.equal(bracket.rounds, 3, "ceil(log2(8)) = 3, at least the floor of 3");
|
| 776 |
assert.equal(bracket.roundsLog.length, 3);
|
| 777 |
for (const { byeName, matches } of bracket.roundsLog) {
|
|
|
|
| 782 |
|
| 783 |
test("Swiss runs strictly fewer pairings than full round-robin for a pool large enough for it to matter", () => {
|
| 784 |
const n = 10;
|
| 785 |
+
const candidates = field(n);
|
| 786 |
+
const bracket = buildSwissBracket(candidates, swissRounds(n), { worlds: ["korrath"], seeds: 1 }, fakePair);
|
| 787 |
const swissPairings = bracket.roundsLog.reduce((a, r) => a + r.matches.length, 0);
|
| 788 |
const roundRobinPairings = (n * (n - 1)) / 2;
|
| 789 |
assert.ok(swissPairings < roundRobinPairings,
|
|
|
|
| 793 |
test("byes rotate: nobody gets a second bye while another candidate hasn't had one yet", () => {
|
| 794 |
// 5 candidates (odd) over enough rounds that everyone gets exactly one bye before anyone gets a
|
| 795 |
// second β 5 rounds guarantees every candidate has been the odd one out exactly once.
|
| 796 |
+
const candidates = field(5);
|
| 797 |
+
const bracket = buildSwissBracket(candidates, 5, { worlds: ["korrath"], seeds: 1 }, fakePair);
|
| 798 |
const byes = bracket.roundsLog.map(r => r.byeName);
|
| 799 |
assert.equal(new Set(byes).size, 5, `every candidate should get exactly one bye across 5 rounds of 5, got: ${byes.join(", ")}`);
|
| 800 |
});
|
|
|
|
| 803 |
// A bye is a candidate the schedule couldn't pair, not a match anyone won β crediting it as a
|
| 804 |
// win (at ANY size) lets a candidate that never fights outrank one that actually won something.
|
| 805 |
// The only thing a bye should move is `byes` (bookkeeping/reporting), never `wins`/`losses`.
|
| 806 |
+
// Pure bookkeeping β no match needs simulating to prove a bye moves only `byes`.
|
| 807 |
+
const opts = { worlds: ["ferros", "vesper"], seeds: 2 };
|
| 808 |
+
const bracket = buildSwissBracket(field(3), swissRounds(3), opts, fakePair);
|
| 809 |
const round = bracket.roundsLog.find(r => r.byeName);
|
| 810 |
assert.ok(round, "fixture: an odd pool must produce a bye somewhere");
|
| 811 |
// byeMatchCount is purely informational β "how big a real pairing this round would have been" β
|
|
|
|
| 820 |
// neverInitiates + a zero standing-army cap (cannot ever field a fight in skirmish) draws a bye,
|
| 821 |
// and a genuine winner must still rank strictly above it β a bye must never be worth what a real
|
| 822 |
// win is worth.
|
| 823 |
+
// Names chosen so the bye lands on a CRIPPLED candidate: pairRound tie-breaks an all-zero round-1
|
| 824 |
+
// standing by name (it used to take whichever was listed last, which made the bye depend on CLI
|
| 825 |
+
// argument order), and the recipient is the last such name. So "zz-crippled" draws it, leaving
|
| 826 |
+
// "a-normal" to actually fight β which is what gives this test something to compare.
|
| 827 |
+
const normal = { name: "a-normal", strategy: "default" };
|
| 828 |
+
const crippled = (name) => ({
|
| 829 |
+
name, strategy: `labByeCrippled_${name}`,
|
| 830 |
+
overrides: { strategies: { [`labByeCrippled_${name}`]: { neverInitiates: true, standingArmyCap: 0 } } },
|
| 831 |
});
|
| 832 |
+
const candidates = [normal, crippled("m-crippled"), crippled("zz-crippled")];
|
|
|
|
|
|
|
| 833 |
const opts = shortSwiss({ worlds: ["korrath"], minutes: 15, seeds: 1, rounds: 1 });
|
| 834 |
const [bracket] = runSwissTournament(candidates, opts);
|
| 835 |
const byeRecipient = bracket.roundsLog[0].byeName;
|
| 836 |
assert.ok(byeRecipient, "fixture: 3 candidates is odd, a bye must fire");
|
| 837 |
const byeRow = bracket.standings.find(s => s.name === byeRecipient);
|
| 838 |
+
const normalRow = bracket.standings.find(s => s.name === "a-normal");
|
| 839 |
assert.equal(byeRow.wins, 0, "a candidate that never fought must have zero wins, bye or not");
|
| 840 |
assert.ok(normalRow.wins > byeRow.wins,
|
| 841 |
`the genuine winner (${normalRow.wins} wins) must rank strictly above a bye recipient that never fought (${byeRow.wins} wins)`);
|
|
|
|
| 849 |
// with the old algorithm). pairRound now backtracks to find a zero-repeat matching whenever one
|
| 850 |
// exists, so this checks across several seed bases β including the ones independently confirmed
|
| 851 |
// to break the old algorithm β rather than trusting a single roll.
|
| 852 |
+
const candidates = field(6);
|
| 853 |
for (const seedBase of [1, 6, 7, 9, 10]) {
|
| 854 |
+
const bracket = buildSwissBracket(candidates, 3, { worlds: ["korrath"], seeds: 1, seedBase }, fakePair);
|
| 855 |
const seen = new Set();
|
| 856 |
for (const { matches } of bracket.roundsLog) {
|
| 857 |
for (const res of matches) {
|
|
|
|
| 905 |
assert.equal(totalWins, totalLosses, "every win must have a matching loss somewhere β a bye must add to neither");
|
| 906 |
});
|
| 907 |
|
| 908 |
+
test("standings are sorted by win RATE, with raw wins as the tie-break", () => {
|
| 909 |
+
// Updated, not deleted: this used to assert most-wins-first, which ranked "everyone who avoided a
|
| 910 |
+
// bye, then everyone who didn't" β a bye recipient plays one fewer pairing and so has strictly
|
| 911 |
+
// fewer chances to earn a win. Rate is the contract now; wins break a rate tie.
|
| 912 |
+
const bracket = buildSwissBracket(field(6), swissRounds(6), { worlds: ["korrath"], seeds: 1 }, fakePair);
|
| 913 |
+
const rate = r => { const n = r.wins + r.losses + r.draws; return n > 0 ? r.wins / n : 0; };
|
| 914 |
for (let i = 1; i < bracket.standings.length; i++) {
|
| 915 |
const prev = bracket.standings[i - 1], cur = bracket.standings[i];
|
| 916 |
+
assert.ok(rate(prev) > rate(cur) || (rate(prev) === rate(cur) && prev.wins >= cur.wins),
|
| 917 |
`standings order violated at index ${i}: ${JSON.stringify(prev)} before ${JSON.stringify(cur)}`);
|
| 918 |
}
|
| 919 |
});
|
|
|
|
| 1001 |
assert.equal(STRATEGIES.aggressive.garrisonMult, 0.4, "the shipped garrisonMult, not a test's override");
|
| 1002 |
assert.equal(STRATEGIES.aggressive.attackTimeoutMult, 0.55);
|
| 1003 |
});
|
| 1004 |
+
|
| 1005 |
+
test("the side-swap is a PAIRED comparison: both directions play the same maps (T2)", () => {
|
| 1006 |
+
// runSwappedDuel's own header says the swap exists so "a side-symmetry bug would show up as those
|
| 1007 |
+
// two disagreeing". It could not: duelSeed hashes the candidate names in ORDER, and the two
|
| 1008 |
+
// directions call runDuel(a,b) then runDuel(b,a), so the halves differed in seat AND MAP.
|
| 1009 |
+
// A disagreement was therefore confounded with ordinary seed variance and carried no information
|
| 1010 |
+
// about seat symmetry β while the seat asymmetry it was meant to catch is real and measurable.
|
| 1011 |
+
// swapAsym, the map-side control, was already paired correctly by replicate parity; this makes
|
| 1012 |
+
// the seat-side control match.
|
| 1013 |
+
const res = runSwappedDuel({ name: "alpha" }, { name: "beta" },
|
| 1014 |
+
{ worlds: ["korrath"], seeds: 2, minutes: 2 });
|
| 1015 |
+
const fixture = rows => rows.map(r => [r.world, r.seed, r.swapAsym]);
|
| 1016 |
+
assert.deepEqual(fixture(res.bAsAi.rows), fixture(res.aAsAi.rows),
|
| 1017 |
+
"both directions must play the SAME world/seed/asym fixtures β only the seat differs");
|
| 1018 |
+
});
|
| 1019 |
+
|
| 1020 |
+
test("the round-1 bye doesn't depend on the order candidates were listed in (T2)", () => {
|
| 1021 |
+
// pairRound sorts on wins alone, and in round 1 every standing is 0-0. V8's sort is stable, so
|
| 1022 |
+
// the order is preserved and the loop takes the LAST element β i.e. `--candidates a,β¦,e`
|
| 1023 |
+
// structurally penalised whichever file was listed last. Nothing to do with merit.
|
| 1024 |
+
const names = ["A", "B", "C", "D", "E"];
|
| 1025 |
+
const rows = order => order.map(n => ({ name: n, wins: 0, losses: 0, draws: 0, byes: 0 }));
|
| 1026 |
+
const byeFor = order => pairRound(rows(order), new Set(), new Set()).byeName;
|
| 1027 |
+
const a = byeFor(names);
|
| 1028 |
+
const b = byeFor([...names].reverse());
|
| 1029 |
+
const c = byeFor(["C", "A", "E", "B", "D"]);
|
| 1030 |
+
assert.equal(a, b, `listing the same field in reverse changed the bye (${a} vs ${b})`);
|
| 1031 |
+
assert.equal(a, c, `listing the same field shuffled changed the bye (${a} vs ${c})`);
|
| 1032 |
+
});
|
| 1033 |
+
|
| 1034 |
+
test("standings rank by RESULT, not by how many pairings the schedule handed out (T2)", () => {
|
| 1035 |
+
// A bye is scorable-neutral (the e9ad1d0 fix) β but the standings still sorted on absolute win
|
| 1036 |
+
// totals, and a bye recipient simply plays one fewer pairing, i.e. worlds x seeds x 2 fewer
|
| 1037 |
+
// chances to earn a win (16 under the swiss CLI defaults). So the ranking was literally "everyone
|
| 1038 |
+
// who avoided a bye, then everyone who didn't": a 2W-2L candidate at 50% ranked below a 3W-3L
|
| 1039 |
+
// candidate at 50% purely for having played more. This is the mirror image of the bug e9ad1d0
|
| 1040 |
+
// just fixed, in the one place the tool is supposed to be trustworthy.
|
| 1041 |
+
const played = { name: "played-more", wins: 3, losses: 3, draws: 0, byes: 0 };
|
| 1042 |
+
const byed = { name: "took-a-bye", wins: 2, losses: 2, draws: 0, byes: 1 };
|
| 1043 |
+
const ranked = rankStandings([played, byed]);
|
| 1044 |
+
assert.equal(ranked[0].name, ranked[1].name === "played-more" ? "took-a-bye" : "played-more",
|
| 1045 |
+
"sanity: the two are distinct rows");
|
| 1046 |
+
assert.deepEqual(ranked.map(r => r.name).sort(), ["played-more", "took-a-bye"]);
|
| 1047 |
+
assert.equal(rankStandings([played, byed])[0].wins / 6, 0.5, "both are at 50% β this is a genuine tie");
|
| 1048 |
+
// The real assertion: equal rates must not be broken in favour of the bigger denominator.
|
| 1049 |
+
const strictlyBetter = { name: "better", wins: 3, losses: 1, draws: 0, byes: 1 };
|
| 1050 |
+
assert.equal(rankStandings([played, strictlyBetter])[0].name, "better",
|
| 1051 |
+
"a 75% record must outrank a 50% one even though it played fewer pairings");
|
| 1052 |
+
});
|
|
@@ -621,8 +621,14 @@ export function pinnedDuelDials(difficulty) {
|
|
| 621 |
// Every duel match's seed is derived from (base seed, world, difficulty, BOTH candidate
|
| 622 |
// names, replicate) β reproducible on its own, same discipline as runSeed() above, just
|
| 623 |
// keyed on the pair rather than a single strategy since a duel row always involves two.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 624 |
const duelSeed = (base, world, difficulty, aName, bName, rep) =>
|
| 625 |
-
hashStr(`${base}:duel:${world}:${difficulty}:${aName
|
| 626 |
|
| 627 |
// Run ONE match: candidate A as owner "player", candidate B as owner "ai", both spending
|
| 628 |
// from the SAME `dials` object (see pinnedDuelDials/FAIRNESS above). `swapAsym` exchanges
|
|
@@ -985,8 +991,29 @@ function greedyMatching(pool, played) {
|
|
| 985 |
// this function stays pure and easy to test on plain data. Exported for direct, fast unit tests
|
| 986 |
// against synthetic standings/played-sets β the pairing algorithm's own correctness doesn't
|
| 987 |
// depend on running real matches.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 988 |
export function pairRound(standingsRows, played, hadBye) {
|
| 989 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 990 |
let byeName = null;
|
| 991 |
if (order.length % 2 === 1) {
|
| 992 |
let idx = -1;
|
|
@@ -1003,7 +1030,13 @@ export function pairRound(standingsRows, played, hadBye) {
|
|
| 1003 |
// fairness. Standings accumulate match wins/losses/draws round over round, same tallying
|
| 1004 |
// runRoundRobinSwapped already does per pairing; a bye adds only to `byes`, never `wins`/
|
| 1005 |
// `losses` β see the header comment above.
|
| 1006 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1007 |
const worlds = opts.worlds || ["korrath", "ferros", "vesper", "kybernet"];
|
| 1008 |
const seeds = opts.seeds ?? 2;
|
| 1009 |
const byeMatchCount = worlds.length * seeds * 2; // informational only β == a real pairing's runSwappedDuel `n`
|
|
@@ -1025,7 +1058,7 @@ function runSwissBracket(candidates, numRounds, opts) {
|
|
| 1025 |
const a = candidates.find(c => c.name === aRow.name);
|
| 1026 |
const b = candidates.find(c => c.name === bRow.name);
|
| 1027 |
played.add(pairKey(a.name, b.name));
|
| 1028 |
-
const res =
|
| 1029 |
const A = standings.get(a.name), B = standings.get(b.name);
|
| 1030 |
A.wins += res.aWins; A.losses += res.bWins; A.draws += res.draws;
|
| 1031 |
B.wins += res.bWins; B.losses += res.aWins; B.draws += res.draws;
|
|
@@ -1033,10 +1066,16 @@ function runSwissBracket(candidates, numRounds, opts) {
|
|
| 1033 |
}
|
| 1034 |
rounds.push({ round, byeName, byeMatchCount: byeName ? byeMatchCount : null, matches });
|
| 1035 |
}
|
| 1036 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1037 |
return {
|
| 1038 |
difficulty: pinnedDuelDials(opts.difficulty).difficulty, // normalized β see pinnedDuelDials' own comment
|
| 1039 |
-
|
| 1040 |
};
|
| 1041 |
}
|
| 1042 |
|
|
|
|
| 621 |
// Every duel match's seed is derived from (base seed, world, difficulty, BOTH candidate
|
| 622 |
// names, replicate) β reproducible on its own, same discipline as runSeed() above, just
|
| 623 |
// keyed on the pair rather than a single strategy since a duel row always involves two.
|
| 624 |
+
// The pair's names are SORTED into the hash, so runDuel(a,b) and runDuel(b,a) draw the identical
|
| 625 |
+
// map. runSwappedDuel exists to isolate seat asymmetry β its header says "a side-symmetry bug would
|
| 626 |
+
// show up as those two disagreeing" β and with an order-dependent seed the two halves differed in
|
| 627 |
+
// seat AND map, so a disagreement was confounded with ordinary seed variance and carried no
|
| 628 |
+
// information at all. (swapAsym, the map-side control, was already paired correctly by replicate
|
| 629 |
+
// parity; this makes the seat-side control match it.)
|
| 630 |
const duelSeed = (base, world, difficulty, aName, bName, rep) =>
|
| 631 |
+
hashStr(`${base}:duel:${world}:${difficulty}:${[aName, bName].sort().join("|")}:${rep}`);
|
| 632 |
|
| 633 |
// Run ONE match: candidate A as owner "player", candidate B as owner "ai", both spending
|
| 634 |
// from the SAME `dials` object (see pinnedDuelDials/FAIRNESS above). `swapAsym` exchanges
|
|
|
|
| 991 |
// this function stays pure and easy to test on plain data. Exported for direct, fast unit tests
|
| 992 |
// against synthetic standings/played-sets β the pairing algorithm's own correctness doesn't
|
| 993 |
// depend on running real matches.
|
| 994 |
+
// Rank a Swiss field. Win RATE, with raw wins as the tie-break β not absolute wins, which is what
|
| 995 |
+
// this used to sort on. A bye is scorable-neutral (nothing added to wins or losses), but its
|
| 996 |
+
// recipient simply plays one fewer pairing, i.e. worlds x seeds x 2 fewer chances to earn a win β
|
| 997 |
+
// 16 under the swiss CLI defaults. Sorting on totals therefore ranked "everyone who avoided a bye,
|
| 998 |
+
// then everyone who didn't": a 2W-2L candidate at 50% finished below a 3W-3L candidate at 50%
|
| 999 |
+
// purely for having played more. That is the mirror image of the bug e9ad1d0 fixed, in the one
|
| 1000 |
+
// place the tool is meant to be trustworthy. Ties fall back to name so the order is total.
|
| 1001 |
+
export function rankStandings(rows) {
|
| 1002 |
+
const rate = r => {
|
| 1003 |
+
const n = (r.wins || 0) + (r.losses || 0) + (r.draws || 0);
|
| 1004 |
+
return n > 0 ? (r.wins || 0) / n : 0;
|
| 1005 |
+
};
|
| 1006 |
+
return [...rows].sort((x, y) =>
|
| 1007 |
+
rate(y) - rate(x) || (y.wins || 0) - (x.wins || 0) || (x.losses || 0) - (y.losses || 0)
|
| 1008 |
+
|| (x.name < y.name ? -1 : x.name > y.name ? 1 : 0));
|
| 1009 |
+
}
|
| 1010 |
+
|
| 1011 |
export function pairRound(standingsRows, played, hadBye) {
|
| 1012 |
+
// Tie-break the pairing order by NAME, not by list position. Every standing is 0-0 in round 1, and
|
| 1013 |
+
// a stable sort preserved input order, so the loop below always took the last-LISTED candidate:
|
| 1014 |
+
// `--candidates a.json,β¦,e.json` structurally penalised e.json for nothing but its position.
|
| 1015 |
+
const order = [...standingsRows].sort((a, b) =>
|
| 1016 |
+
b.wins - a.wins || (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));
|
| 1017 |
let byeName = null;
|
| 1018 |
if (order.length % 2 === 1) {
|
| 1019 |
let idx = -1;
|
|
|
|
| 1030 |
// fairness. Standings accumulate match wins/losses/draws round over round, same tallying
|
| 1031 |
// runRoundRobinSwapped already does per pairing; a bye adds only to `byes`, never `wins`/
|
| 1032 |
// `losses` β see the header comment above.
|
| 1033 |
+
// The Swiss SCHEDULE, with the match runner injected. Extracted so the combinatorics β round count,
|
| 1034 |
+
// pairings per round, bye rotation, rematch avoidance, standings order β can be tested on synthetic
|
| 1035 |
+
// results in milliseconds instead of by simulating real 15-40 sim-minute matches. Those schedule
|
| 1036 |
+
// tests were roughly 50 s of the suite's wall clock, all of it spent proving arithmetic. (The file
|
| 1037 |
+
// already demonstrated the fast idiom once, in pairRound's own test, and said why it was better.)
|
| 1038 |
+
// runSwissBracket below is now a thin wrapper that injects the real runSwappedDuel.
|
| 1039 |
+
export function buildSwissBracket(candidates, numRounds, opts, runPair) {
|
| 1040 |
const worlds = opts.worlds || ["korrath", "ferros", "vesper", "kybernet"];
|
| 1041 |
const seeds = opts.seeds ?? 2;
|
| 1042 |
const byeMatchCount = worlds.length * seeds * 2; // informational only β == a real pairing's runSwappedDuel `n`
|
|
|
|
| 1058 |
const a = candidates.find(c => c.name === aRow.name);
|
| 1059 |
const b = candidates.find(c => c.name === bRow.name);
|
| 1060 |
played.add(pairKey(a.name, b.name));
|
| 1061 |
+
const res = runPair(a, b, { ...opts, seedBase: roundSeedBase });
|
| 1062 |
const A = standings.get(a.name), B = standings.get(b.name);
|
| 1063 |
A.wins += res.aWins; A.losses += res.bWins; A.draws += res.draws;
|
| 1064 |
B.wins += res.bWins; B.losses += res.aWins; B.draws += res.draws;
|
|
|
|
| 1066 |
}
|
| 1067 |
rounds.push({ round, byeName, byeMatchCount: byeName ? byeMatchCount : null, matches });
|
| 1068 |
}
|
| 1069 |
+
return {
|
| 1070 |
+
rounds: numRounds, byeMatchCount, roundsLog: rounds,
|
| 1071 |
+
standings: rankStandings([...standings.values()]),
|
| 1072 |
+
};
|
| 1073 |
+
}
|
| 1074 |
+
|
| 1075 |
+
function runSwissBracket(candidates, numRounds, opts) {
|
| 1076 |
return {
|
| 1077 |
difficulty: pinnedDuelDials(opts.difficulty).difficulty, // normalized β see pinnedDuelDials' own comment
|
| 1078 |
+
...buildSwissBracket(candidates, numRounds, opts, runSwappedDuel),
|
| 1079 |
};
|
| 1080 |
}
|
| 1081 |
|
|
@@ -78,11 +78,28 @@ export function createSelfPlayState({
|
|
| 78 |
}
|
| 79 |
|
| 80 |
/**
|
| 81 |
-
* Advance a self-play state by one fixed step
|
| 82 |
-
*
|
| 83 |
-
*
|
| 84 |
-
*
|
| 85 |
-
*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
* @param {State} state @param {number} [dt]
|
| 87 |
*/
|
| 88 |
export function tickSelfPlay(state, dt = DT) {
|
|
|
|
| 78 |
}
|
| 79 |
|
| 80 |
/**
|
| 81 |
+
* Advance a self-play state by one fixed step: owner "player" is driven explicitly here, then
|
| 82 |
+
* tick(state, dt) runs its own hardcoded runAI(state, dt) for owner "ai" (engine/sim.js).
|
| 83 |
+
*
|
| 84 |
+
* ORDERING, STATED HONESTLY. This comment used to claim both controllers think on the IDENTICAL
|
| 85 |
+
* pre-tick snapshot. They do not. runAI(β¦, "player") is not a read: issueBuild inserts into
|
| 86 |
+
* state.buildings immediately, queueProduction mutates queues, payCost mutates resources. Both
|
| 87 |
+
* controllers share THINK_INTERVAL and start their countdowns together, so the "ai" seat reads
|
| 88 |
+
* state the "player" seat has already changed that frame. Measured over 400 player think-cycles in
|
| 89 |
+
* a 10-sim-minute korrath match: orders differed on 13% of cycles, queues and resources on 6%, and
|
| 90 |
+
* buildings on 1%.
|
| 91 |
+
*
|
| 92 |
+
* The channel that actually feeds decisions is `buildings` β chooseAttackTarget's seenBuildings
|
| 93 |
+
* scan and counterToPlayerArmy's static-defense scan both walk state.buildings.values() β so a
|
| 94 |
+
* turret the player controller founded microseconds earlier is answerable one think cycle (1.5 s)
|
| 95 |
+
* sooner than the mirror case ever could be. Small, but FIXED IN DIRECTION: it always favours the
|
| 96 |
+
* "ai" seat, every cycle, in every duel and Swiss match.
|
| 97 |
+
*
|
| 98 |
+
* Why it isn't simply fixed here: making the claim true means either a genuinely frozen pre-tick
|
| 99 |
+
* read, or moving the "ai" call out of sim.js's hardcoded pipeline so the two can alternate. Both
|
| 100 |
+
* restructure engine/sim.js's tick contract, which is a larger change than this seam. The bound is
|
| 101 |
+
* pinned by a characterisation test in test/ai-selfplay.test.js instead, so the edge cannot grow
|
| 102 |
+
* unnoticed while the real fix is scheduled (docs/code-improvement-tiers.md, Tier 3).
|
| 103 |
* @param {State} state @param {number} [dt]
|
| 104 |
*/
|
| 105 |
export function tickSelfPlay(state, dt = DT) {
|