Update logbook: Repro - NonZero: Interaction-Guided Exploration for Multi-Agent Monte Carlo Tree Search

logbook.json

@@ -10,7 +10,7 @@
     "icml2026-repro",
     "paper-Jh6gq9QsFa"
   ],
-  "updated_at": "2026-08-03T03:17:41+00:00",
+  "updated_at": "2026-08-03T03:18:42+00:00",
   "root": {
     "slug": "index",
     "title": "Repro - NonZero: Interaction-Guided Exploration for Multi-Agent Monte Carlo Tree Search",
@@ -73,10 +73,10 @@
     "total_size": 0,
     "bucket_id": null
   },
-  "agent_view_tokens": 3382,
+  "agent_view_tokens": 3525,
   "trace_view_tokens": 10,
   "workspace_view_tokens": 8,
-  "revision": "c01b16b6a6c366d50096",
+  "revision": "319798aa4827526a64f7",
   "workspace_ref": {
     "repo_id": "byte-vortex/nonzero-repro-artifacts",
     "repo_type": "bucket",

pages/claim-4-matgame-8-agent-10-action-nonlinear/page.md

@@ -6,3 +6,317 @@
 {"type": "markdown", "id": "cell_bc24ae0bf5e1", "created_at": "2026-08-03T03:15:00+00:00", "title": "[toy / partial] Built a generalized N-agent MatGame environment implementing th…"}
 -->
 **[toy / partial]** Built a generalized N-agent MatGame environment implementing the exact reward rule from Appendix D (linear = sum of agent action indices; nonlinear = linear sum + Gaussian(0,2^2) + Uniform(-3,3) noise). Sanity-checked reward scale against Table 1: random-policy episodes on the 8x10 nonlinear config average ~370-380 out of a 720 max, consistent with the paper's trained-baseline range of 587-712 sitting well above random. Confirmed the underlying MAZero codebase (which NonZero extends, per the paper's own Appendix C) builds and trains successfully on a single Kaggle T4 GPU, converging to the optimal joint policy (test score 100.0 +/- 0.0) on the repo's built-in matgame3 config. Began implementing NonUCT's selection rule as a C++ patch to MAZero's MCTS node scoring (cnode.h/cnode.cpp): added hooks for a precomputed surrogate score, blended additively into the existing pUCT formula, gated by a weight defaulting to 0 so baseline behavior is unaffected unless explicitly enabled. This patch compiles successfully. **Not completed**: the Cython bindings (cytree.pyx) needed to call these new C++ hooks from Python were not implemented, so no NonZero-vs-MAZero training comparison was run, and Table 1's actual numbers (697.1 vs 672.3) were not reproduced or falsified.
+
+
+---
+<!-- trackio-cell
+{"type": "code", "id": "cell_e6b23a9eda0a", "created_at": "2026-08-03T03:18:41+00:00", "title": "Run: hf claim4_toy_comparison.py (exit 0)", "command": ["hf", "jobs", "uv", "run", "--flavor", "cpu-basic", "--secrets", "HF_TOKEN", "claim4_toy_comparison.py"], "exit_code": 0, "duration_s": 55.853}
+-->
+````bash
+$ hf jobs uv run --flavor cpu-basic --secrets HF_TOKEN claim4_toy_comparison.py
+````
+
+exit 0 · 55.9s
+
+
+````python title=claim4_toy_comparison.py
+
+# /// script
+# dependencies = ["torch", "numpy"]
+# ///
+"""
+Claim 4 -- toy-scale head-to-head comparison on the paper's exact
+8-agent, 10-action nonlinear MatGame setting (Table 1). Vectorized
+version: candidate generation uses precomputed batched tensor masks
+instead of per-candidate Python loops, since the naive version is too
+slow for 2000 decisions.
+"""
+
+import torch
+import torch.nn as nn
+import numpy as np
+
+
+class MatgameNonZeroEnv:
+    def __init__(self, n_agents=8, n_actions=10, reward_type="nonlinear",
+                 noise_std=2.0, noise_uniform_range=3.0, seed=None):
+        self.n_agents = n_agents
+        self.n_actions = n_actions
+        self.reward_type = reward_type
+        self.noise_std = noise_std
+        self.noise_uniform_range = noise_uniform_range
+        self._rng = np.random.default_rng(seed)
+
+    def joint_reward(self, actions):
+        base = float(np.sum(actions))
+        if self.reward_type == "linear":
+            return base
+        u = self._rng.normal(0.0, self.noise_std)
+        v = self._rng.uniform(-self.noise_uniform_range, self.noise_uniform_range)
+        return base + u + v
+
+
+class VectorizedProposer:
+    def __init__(self, n_agents, n_actions, c=1.0, alpha=1.0):
+        self.n_agents, self.n_actions = n_agents, n_actions
+        self.nd = n_agents * n_actions
+        self.c, self.alpha = c, alpha
+
+        rows1 = n_agents * n_actions
+        self.agent1 = torch.zeros(rows1, dtype=torch.long)
+        self.action1 = torch.zeros(rows1, dtype=torch.long)
+        self.block_mask1 = torch.zeros(rows1, self.nd)
+        self.target1 = torch.zeros(rows1, self.nd)
+        r = 0
+        for i in range(n_agents):
+            for j in range(n_actions):
+                self.agent1[r], self.action1[r] = i, j
+                self.block_mask1[r, i * n_actions:(i + 1) * n_actions] = 1.0
+                self.target1[r, i * n_actions + j] = 1.0
+                r += 1
+
+        pairs = [(i, j) for i in range(n_agents) for j in range(i + 1, n_agents)]
+        rows2 = len(pairs) * n_actions * n_actions
+        self.pair_i = torch.zeros(rows2, dtype=torch.long)
+        self.pair_j = torch.zeros(rows2, dtype=torch.long)
+        self.action_i = torch.zeros(rows2, dtype=torch.long)
+        self.action_j = torch.zeros(rows2, dtype=torch.long)
+        self.block_mask2 = torch.zeros(rows2, self.nd)
+        self.target2 = torch.zeros(rows2, self.nd)
+        r = 0
+        for (i, j) in pairs:
+            for ai in range(n_actions):
+                for aj in range(n_actions):
+                    self.pair_i[r], self.pair_j[r] = i, j
+                    self.action_i[r], self.action_j[r] = ai, aj
+                    self.block_mask2[r, i * n_actions:(i + 1) * n_actions] = 1.0
+                    self.block_mask2[r, j * n_actions:(j + 1) * n_actions] = 1.0
+                    self.target2[r, i * n_actions + ai] = 1.0
+                    self.target2[r, j * n_actions + aj] = 1.0
+                    r += 1
+
+    def one_hot(self, agent_actions):
+        agent_actions = torch.as_tensor(agent_actions, dtype=torch.long)
+        oh = torch.nn.functional.one_hot(agent_actions, num_classes=self.n_actions)
+        return oh.reshape(-1).float()
+
+    def eta(self, theta, a_batch):
+        return self.c * torch.asinh(self.alpha * (theta.unsqueeze(0) * a_batch).sum(dim=-1))
+
+    def best_candidate(self, theta, cur_enc, current_actions):
+        cur_eta = self.eta(theta, cur_enc.unsqueeze(0)).item()
+
+        cand1 = cur_enc.unsqueeze(0) * (1 - self.block_mask1) + self.target1
+        eta1 = self.eta(theta, cand1)
+        current_action1 = torch.as_tensor(current_actions)[self.agent1]
+        self_mask1 = (self.action1 == current_action1)
+        eta1 = eta1.masked_fill(self_mask1, -float("inf"))
+        delta1 = eta1 - cur_eta
+        best1_val, best1_idx = delta1.max(dim=0)
+
+        cand2 = cur_enc.unsqueeze(0) * (1 - self.block_mask2) + self.target2
+        eta2 = self.eta(theta, cand2)
+        cur_ai = torch.as_tensor(current_actions)[self.pair_i]
+        cur_aj = torch.as_tensor(current_actions)[self.pair_j]
+        self_mask2 = (self.action_i == cur_ai) | (self.action_j == cur_aj)
+        eta2 = eta2.masked_fill(self_mask2, -float("inf"))
+        delta2 = eta2 - cur_eta
+        best2_val, best2_idx = delta2.max(dim=0)
+
+        if best1_val.item() >= best2_val.item():
+            agent_idx, new_action = self.agent1[best1_idx].item(), self.action1[best1_idx].item()
+            candidate_actions = current_actions.copy()
+            candidate_actions[agent_idx] = new_action
+        else:
+            i, j = self.pair_i[best2_idx].item(), self.pair_j[best2_idx].item()
+            ai, aj = self.action_i[best2_idx].item(), self.action_j[best2_idx].item()
+            candidate_actions = current_actions.copy()
+            candidate_actions[i], candidate_actions[j] = ai, aj
+
+        return candidate_actions
+
+
+def fit_surrogate(env, n_agents, n_actions, n_samples=4000, n_epochs=1500, rng=None):
+    actions = rng.integers(0, n_actions, size=(n_samples, n_agents))
+    rewards = torch.tensor([env.joint_reward(a) for a in actions], dtype=torch.float32)
+    onehots = torch.nn.functional.one_hot(torch.tensor(actions, dtype=torch.long), num_classes=n_actions)
+    a_enc = onehots.reshape(n_samples, -1).float()
+
+    nd = n_agents * n_actions
+    theta = nn.Parameter(torch.zeros(nd))
+    log_c, log_alpha = nn.Parameter(torch.zeros(())), nn.Parameter(torch.zeros(()))
+    opt = torch.optim.Adam([theta, log_c, log_alpha], lr=0.05)
+    sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=n_epochs)
+
+    for _ in range(n_epochs):
+        opt.zero_grad()
+        pred = torch.exp(log_c) * torch.asinh(torch.exp(log_alpha) * (theta * a_enc).sum(dim=-1))
+        loss = ((pred - rewards) ** 2).mean()
+        loss.backward()
+        opt.step()
+        sched.step()
+    return theta.detach()
+
+
+def nonzero_style_decision(env, proposer, theta, budget, rng):
+    n_agents, n_actions = env.n_agents, env.n_actions
+    current = rng.integers(0, n_actions, size=n_agents)
+    best_reward = env.joint_reward(current)
+    evals_used = 1
+
+    while evals_used < budget:
+        cur_enc = proposer.one_hot(current)
+        candidate_actions = proposer.best_candidate(theta, cur_enc, current)
+        real_reward = env.joint_reward(candidate_actions)
+        evals_used += 1
+
+        if real_reward > best_reward:
+            current = candidate_actions
+            best_reward = real_reward
+        else:
+            break
+
+    return best_reward
+
+
+def mazero_style_decision(env, budget, rng):
+    best_reward = -float("inf")
+    for _ in range(budget):
+        actions = rng.integers(0, env.n_actions, size=env.n_agents)
+        r = env.joint_reward(actions)
+        if r > best_reward:
+            best_reward = r
+    return best_reward
+
+
+def main():
+    print("=" * 70)
+    print("NonZero reproduction -- Claim 4 toy comparison (vectorized)")
+    print("Setting: 8 agents, 10 actions, nonlinear (Table 1's hardest config)")
+    print("2000 sequential decisions, equal real-eval budget per decision")
+    print("=" * 70)
+
+    n_agents, n_actions = 8, 10
+    n_steps = 2000
+    budget_per_step = 15
+
+    rng_fit = np.random.default_rng(0)
+    env_fit = MatgameNonZeroEnv(n_agents=n_agents, n_actions=n_actions,
+                                 reward_type="nonlinear", seed=1)
+    theta = fit_surrogate(env_fit, n_agents, n_actions, rng=rng_fit)
+    proposer = VectorizedProposer(n_agents, n_actions)
+    print(f"\nSurrogate fit complete. Candidate masks precomputed "
+          f"({proposer.block_mask1.shape[0]} first-order + "
+          f"{proposer.block_mask2.shape[0]} second-order candidates/decision).")
+
+    nonzero_rewards, mazero_rewards = [], []
+    for step in range(n_steps):
+        env_a = MatgameNonZeroEnv(n_agents=n_agents, n_actions=n_actions,
+                                   reward_type="nonlinear", seed=5000 + step)
+        rng_a = np.random.default_rng(3000 + step)
+        nonzero_rewards.append(nonzero_style_decision(env_a, proposer, theta, budget_per_step, rng_a))
+
+        env_b = MatgameNonZeroEnv(n_agents=n_agents, n_actions=n_actions,
+                                   reward_type="nonlinear", seed=5000 + step)
+        rng_b = np.random.default_rng(3000 + step)
+        mazero_rewards.append(mazero_style_decision(env_b, budget_per_step, rng_b))
+
+        if (step + 1) % 400 == 0:
+            print(f"  ...{step + 1}/{n_steps} decisions completed")
+
+    nz_mean, nz_std = np.mean(nonzero_rewards), np.std(nonzero_rewards)
+    mz_mean, mz_std = np.mean(mazero_rewards), np.std(mazero_rewards)
+    improvement = (nz_mean - mz_mean) / abs(mz_mean) * 100
+
+    print(f"\n[Results over {n_steps} decisions, budget={budget_per_step}/decision]")
+    print(f"  NonZero-style (guided proposal):  {nz_mean:.2f} +/- {nz_std:.2f}")
+    print(f"  MAZero-style (random sampled):    {mz_mean:.2f} +/- {mz_std:.2f}")
+    print(f"  Improvement: {improvement:+.1f}%")
+
+    print("\n" + "=" * 70)
+    print("SUMMARY")
+    if nz_mean > mz_mean:
+        print(f"  PASS (toy scale): NonZero-style guided proposal outperforms")
+        print(f"  MAZero-style random sampling by {improvement:.1f}% under equal budget.")
+    else:
+        print(f"  FAIL: guided proposal did NOT outperform the baseline.")
+    print("=" * 70)
+
+
+if __name__ == "__main__":
+    main()
+
+````
+
+
+````output
+/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_experimental.py:59: UserWarning: 'HfApi.run_uv_job' is experimental and might be subject to breaking changes in the future without prior notice. You can disable this warning by setting `HF_HUB_DISABLE_EXPERIMENTAL_WARNING=1` as environment variable.
+  warnings.warn(
+Job started with ID: 6a70085a6b79c09949c20239
+View at: https://huggingface.co/jobs/byte-vortex/6a70085a6b79c09949c20239
+Downloading nvidia-cuda-runtime (2.1MiB)
+Downloading nvidia-cudnn-cu13 (349.2MiB)
+Downloading nvidia-nvjitlink (38.9MiB)
+Downloading numpy (15.9MiB)
+Downloading nvidia-cuda-cupti (10.2MiB)
+Downloading nvidia-nvshmem-cu13 (57.6MiB)
+Downloading torch (502.2MiB)
+Downloading nvidia-cublas (403.5MiB)
+Downloading sympy (6.0MiB)
+Downloading nvidia-cusolver (191.6MiB)
+Downloading triton (188.6MiB)
+Downloading nvidia-cusparse (139.2MiB)
+Downloading nvidia-cuda-nvrtc (86.0MiB)
+Downloading nvidia-cusparselt-cu13 (162.3MiB)
+Downloading nvidia-curand (56.8MiB)
+Downloading nvidia-cufile (1.2MiB)
+Downloading nvidia-nccl-cu13 (196.4MiB)
+Downloading cuda-bindings (6.3MiB)
+Downloading nvidia-cufft (204.2MiB)
+Downloading networkx (2.0MiB)
+ Downloaded nvidia-cufile
+ Downloaded nvidia-cuda-runtime
+ Downloaded networkx
+ Downloaded cuda-bindings
+ Downloaded nvidia-cuda-cupti
+ Downloaded numpy
+ Downloaded nvidia-nvjitlink
+ Downloaded sympy
+ Downloaded nvidia-curand
+ Downloaded nvidia-nvshmem-cu13
+ Downloaded nvidia-cuda-nvrtc
+ Downloaded nvidia-cusparse
+ Downloaded nvidia-cusparselt-cu13
+ Downloaded nvidia-cusolver
+ Downloaded nvidia-nccl-cu13
+ Downloaded nvidia-cufft
+ Downloaded triton
+ Downloaded nvidia-cudnn-cu13
+ Downloaded nvidia-cublas
+ Downloaded torch
+Installed 30 packages in 233ms
+======================================================================
+NonZero reproduction -- Claim 4 toy comparison (vectorized)
+Setting: 8 agents, 10 actions, nonlinear (Table 1's hardest config)
+2000 sequential decisions, equal real-eval budget per decision
+======================================================================
+
+Surrogate fit complete. Candidate masks precomputed (80 first-order + 2800 second-order candidates/decision).
+  ...400/2000 decisions completed
+  ...800/2000 decisions completed
+  ...1200/2000 decisions completed
+  ...1600/2000 decisions completed
+  ...2000/2000 decisions completed
+
+[Results over 2000 decisions, budget=15/decision]
+  NonZero-style (guided proposal):  72.97 +/- 2.59
+  MAZero-style (random sampled):    50.84 +/- 4.51
+  Improvement: +43.5%
+
+======================================================================
+SUMMARY
+  PASS (toy scale): NonZero-style guided proposal outperforms
+  MAZero-style random sampling by 43.5% under equal budget.
+======================================================================
+
+````

workspace.json

@@ -14,6 +14,11 @@
       "type": "Jobs",
       "label": "byte-vortex/6a7007c56b79c09949c20232"
     },
+    {
+      "url": "https://huggingface.co/jobs/byte-vortex/6a70085a6b79c09949c20239",
+      "type": "Jobs",
+      "label": "byte-vortex/6a70085a6b79c09949c20239"
+    },
     {
       "url": "https://huggingface.co/buckets/byte-vortex/nonzero-repro-artifacts",
       "type": "Buckets",