Wave 21: Stage-0 dataset pipeline — swesmith engine, rollout harness, gates, contract

Implements research/deepread/13-synthesis-architecture.md (ADR-016): the
"point at a repo -> dataset" pipeline the critical review architected, closing
every verified P0 from research/deepread/12-verified-findings.md.

New modules (+96 tests, full suite 511 passed / 66 skipped):
- datagen/repo_gate.py: SPDX-ish license detection -> 3 tiers (fail-closed) +
benchmark DECONTAMINATION vs the SWE-bench eval-repo list (closes V3 — zero
decontamination existed anywhere).
- datagen/swesmith_adapter.py + [swesmith] extra: SWE-smith as the synthesis
engine (closes V4 buy-vs-build). Handles the patch-semantics INVERSION
(SWE-smith's patch INTRODUCES the bug; golden_diff = reverse_unified_diff)
+ strategy provenance sidecar + difficulty priors.
- datagen/trajectory.py: canonical trajectory IR (closes D-11). ToolCall
canonical_form = v1 divergence-gate action algebra (replaces the whitespace
stub, D-3); to_policy_row = THE policy-visible serializer, sentinel-tested
to never leak golden_diff/deleted_symbols (D-8).
- datagen/rollout_harness.py: collect_trajectory agent loop over
FeatureDeletionEnv (closes V2 — the SFT corpus finally has a producer; its
env-grounded episodes are also the tree's seeds, fixing D-1) + typed
admission routing (sft/dpo-candidate/quarantine).
- pipeline/{s3_contract,dedup,build_corpus}.py: ONE reconciled dataset layout
(supersedes F1/F2's divergent contracts, V8) with restricted tasks_full
prefix + golden_diff sha256; stable-hash MinHash dedup incl.
cross-generation signatures (D-12); local write-once stage-driver with
holdout-first split + budget ceiling (D-9/D-21).

Fidelity corrections (verified findings, applied to code+docs):
- V1: SDPO "mathematically the same" claim corrected (opsd.py, mapping doc) —
Cursor cites SDPO as background; ours is a third blog-inspired design.
- V5: fabricated numbers struck/tagged (69.3%, "24 generators", "85% compute").
- V7: Streaming DiLoCo citation fixed (Douillard 2501.18512; Kale 2502.12996).
- V11: teacher_replay cost docstring relabeled (synthetic-trace basis).
- V13: kl_in_reward verl wording (default/recommended, not only).
- ADR-016 records the decision + gates; harvested repo_gate from the one
surviving worktree builder (3 of 4 stalled on API capacity; rebuilt directly).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

composer_replication/datagen/repo_gate.py

@@ -0,0 +1,361 @@
+"""repo_gate.py — Stage-0 ingest gate: license tiers + benchmark decontamination.
+
+Architecture step 1 of the dataset pipeline (research/deepread/
+13-synthesis-architecture.md Part B). Closes two verified findings:
+
+  * V3 / D-5  — ZERO benchmark decontamination existed anywhere in code or
+    designs, while the pipeline trains on SWE-bench-family substrates and is
+    scored on SWE-bench Verified. ``is_eval_contaminated`` is the hard wall:
+    a repo on the eval list is NEVER admitted, regardless of license.
+  * V9 / D-13 — the only license filter was a lowercase substring match on a
+    task field (``substrates.py`` ``is_redistributable``), with no SPDX
+    detection at the repo-ingest path and no trainable-vs-redistributable
+    split. ``detect_license`` + ``license_tier`` replace the boolean with a
+    three-tier verdict.
+
+Why tiers, not a boolean (D-13): weak-copyleft repos (MPL/LGPL) are fine to
+*train on* but we must not *redistribute* derivative diffs from them — a
+boolean "redistributable?" gate either over-excludes them or leaks them into
+published corpora. The tier travels with the verdict so downstream corpus
+steps (step 6) can route TRAINABLE_ONLY rows away from any published split.
+
+Why title-anchored matching for the GNU family: GPL-3.0 §13 mentions the
+"GNU Affero General Public License" by name and AGPL-3.0 §13 mentions the
+"GNU General Public License" — naive full-body substring matching
+misclassifies one as the other. We therefore classify the GNU/MPL/Apache
+family from the document HEADER (first ~400 normalized chars, where the
+license title lives) and only use full-body phrases for the short permissive
+licenses whose titles are not distinctive (MIT/ISC/BSD/Unlicense).
+
+Stdlib-only on purpose: the gate must run before anything heavy is installed.
+"""
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass, field
+from enum import Enum
+from pathlib import Path
+
+# ---------------------------------------------------------------------------
+# License detection (V9 / D-13)
+# ---------------------------------------------------------------------------
+
+#: License files checked in order; first match wins (case-insensitive on name).
+_LICENSE_FILENAMES: tuple[str, ...] = ("LICENSE", "LICENSE.txt", "LICENSE.md", "COPYING")
+
+#: Trove classifier / PEP 639 expression fragments → SPDX id. Secondary signal
+#: only — the classifier cannot distinguish BSD-2 from BSD-3, so it maps to
+#: BSD-3-Clause (the common case) and the LICENSE file is preferred when present.
+_CLASSIFIER_MAP: tuple[tuple[str, str], ...] = (
+    ("gnu affero general public license", "AGPL-3.0"),
+    ("gnu lesser general public license v3", "LGPL-3.0"),
+    ("gnu lesser general public license v2.1", "LGPL-2.1"),
+    ("gnu lesser general public license", "LGPL-3.0"),
+    ("gnu general public license v3", "GPL-3.0"),
+    ("gnu general public license v2", "GPL-2.0"),
+    ("mozilla public license 2.0", "MPL-2.0"),
+    ("apache software license", "Apache-2.0"),
+    ("mit license", "MIT"),
+    ("bsd license", "BSD-3-Clause"),
+    ("isc license", "ISC"),
+    ("the unlicense", "Unlicense"),
+)
+
+#: Bare SPDX ids accepted from PEP 639 ``license = "<expr>"`` in pyproject.
+_SPDX_IDS: frozenset[str] = frozenset(
+    {
+        "MIT", "Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "ISC",
+        "GPL-2.0", "GPL-3.0", "AGPL-3.0", "LGPL-2.1", "LGPL-3.0",
+        "MPL-2.0", "Unlicense",
+    }
+)
+_SPDX_LOOKUP: dict[str, str] = {s.lower(): s for s in _SPDX_IDS}
+# Common -only/-or-later suffixed forms normalize to the base id we tier on.
+for _base in ("GPL-2.0", "GPL-3.0", "AGPL-3.0", "LGPL-2.1", "LGPL-3.0"):
+    _SPDX_LOOKUP[f"{_base.lower()}-only"] = _base
+    _SPDX_LOOKUP[f"{_base.lower()}-or-later"] = _base
+
+
+@dataclass(frozen=True)
+class LicenseInfo:
+    """Outcome of license detection: SPDX-ish id + which signal decided it."""
+
+    spdx_id: str  # one of _SPDX_IDS or "unknown"
+    signal: str  # "license_file" | "classifier" | "none"
+    source: str = ""  # filename that supplied the winning signal
+
+
+def _normalize_text(text: str) -> str:
+    return re.sub(r"\s+", " ", text).strip().lower()
+
+
+#: Title strings for the families that cross-cite each other. NOTE: "gnu
+#: affero general public license" does NOT contain "gnu general public
+#: license" as a substring ("affero" splits it), so the titles are disjoint.
+_HEADER_TITLES: tuple[tuple[str, str], ...] = (
+    ("gnu affero general public license", "agpl"),
+    ("gnu lesser general public license", "lgpl"),
+    ("gnu general public license", "gpl"),
+    ("mozilla public license", "mpl"),
+    ("apache license", "apache"),
+)
+
+
+def _classify_header(header: str) -> str | None:
+    """Title-anchored families (GNU/MPL/Apache). The EARLIEST-occurring title
+    wins, because a license document's own title always precedes any
+    cross-citation — GPL-3 §13 names the AGPL and AGPL-3 §13 names the GPL,
+    so mere presence-matching misclassifies one as the other (the V9 trap)."""
+    hits = [(idx, family) for title, family in _HEADER_TITLES if (idx := header.find(title)) >= 0]
+    if not hits:
+        return None
+    family = min(hits)[1]
+    if family == "agpl":
+        return "AGPL-3.0"
+    if family == "lgpl":
+        return "LGPL-2.1" if "version 2.1" in header else "LGPL-3.0"
+    if family == "gpl":
+        return "GPL-2.0" if "version 2" in header and "version 3" not in header else "GPL-3.0"
+    if family == "mpl":
+        return "MPL-2.0" if "2.0" in header else None
+    return "Apache-2.0" if "version 2.0" in header else None
+
+
+def _classify_body(body: str) -> str | None:
+    """Distinctive-phrase matching for the short permissive licenses. Order
+    matters: ISC's grant ("permission to use, copy, modify") is checked via
+    its unique "and/or distribute … with or without fee" wording so it can't
+    be shadowed by MIT's "permission is hereby granted" phrase."""
+    if "free and unencumbered software released into the public domain" in body:
+        return "Unlicense"
+    # Apache boilerplate notice files ("Licensed under the Apache License,
+    # Version 2.0") carry the title mid-body, not in a header — the tricky
+    # Apache-vs-MIT case: both say "permission"/"license", only Apache names
+    # itself with a version.
+    if "apache license" in body and "version 2.0" in body:
+        return "Apache-2.0"
+    if "permission is hereby granted, free of charge, to any person obtaining a copy" in body:
+        return "MIT"
+    if "with or without fee" in body and "permission to use, copy, modify" in body:
+        return "ISC"
+    if "redistribution and use in source and binary forms" in body:
+        # The third clause ("Neither the name of …") is what separates 3- from 2-.
+        return "BSD-3-Clause" if "neither the name of" in body else "BSD-2-Clause"
+    return None
+
+
+def _classify_license_text(text: str) -> str:
+    norm = _normalize_text(text)
+    return _classify_header(norm[:400]) or _classify_body(norm) or "unknown"
+
+
+def _classifier_signal(repo_root: Path) -> tuple[str, str] | None:
+    """Secondary signal: trove classifiers / PEP 639 license expression in
+    pyproject.toml or setup.py. Regex-scan, not a TOML parse — the gate must
+    not depend on packaging libs and classifiers are line-shaped in practice."""
+    for name in ("pyproject.toml", "setup.py"):
+        path = repo_root / name
+        if not path.is_file():
+            continue
+        try:
+            text = path.read_text(encoding="utf-8", errors="replace")
+        except OSError:
+            continue
+        # PEP 639: license = "Apache-2.0" (pyproject only, but harmless on setup.py).
+        m = re.search(r'license\s*=\s*["\']([A-Za-z0-9.+-]+)["\']', text)
+        if m and m.group(1).lower() in _SPDX_LOOKUP:
+            return _SPDX_LOOKUP[m.group(1).lower()], name
+        low = _normalize_text(text)
+        for fragment, spdx in _CLASSIFIER_MAP:
+            if f"license :: osi approved :: {fragment}" in low or (
+                "license ::" in low and fragment in low
+            ):
+                return spdx, name
+    return None
+
+
+def detect_license(repo_root: Path) -> LicenseInfo:
+    """Detect the repo license. LICENSE-file text is the primary signal;
+    packaging classifiers are secondary (used only when the file is absent or
+    unclassifiable). The winning signal is recorded so corpus manifests can
+    show provenance for the tier decision (V9 closure must be auditable)."""
+    for name in _LICENSE_FILENAMES:
+        path = repo_root / name
+        if not path.is_file():
+            # Case-insensitive fallback (e.g. "License.md", "COPYING.txt" not
+            # matched here on purpose — only exact-name case variants).
+            matches = [p for p in repo_root.glob("*") if p.is_file() and p.name.lower() == name.lower()]
+            path = matches[0] if matches else path
+        if path.is_file():
+            try:
+                text = path.read_text(encoding="utf-8", errors="replace")
+            except OSError:
+                continue
+            spdx = _classify_license_text(text)
+            if spdx != "unknown":
+                return LicenseInfo(spdx_id=spdx, signal="license_file", source=path.name)
+            # File exists but unclassifiable → let the classifier signal try
+            # before giving up; remember we saw a file for the "none" case.
+            fallback = _classifier_signal(repo_root)
+            if fallback is not None:
+                return LicenseInfo(spdx_id=fallback[0], signal="classifier", source=fallback[1])
+            return LicenseInfo(spdx_id="unknown", signal="license_file", source=path.name)
+    fallback = _classifier_signal(repo_root)
+    if fallback is not None:
+        return LicenseInfo(spdx_id=fallback[0], signal="classifier", source=fallback[1])
+    return LicenseInfo(spdx_id="unknown", signal="none")
+
+
+# ---------------------------------------------------------------------------
+# License tiers (D-13: tiers, not a boolean)
+# ---------------------------------------------------------------------------
+
+
+class Tier(Enum):
+    """Three-way license verdict. TRAINABLE_ONLY exists because weak copyleft
+    (MPL/LGPL) permits training but redistribution of derivative diffs would
+    trigger copyleft obligations — collapsing this to a boolean either loses
+    training data or leaks copyleft material into published corpora (D-13)."""
+
+    REDISTRIBUTABLE = "redistributable"
+    TRAINABLE_ONLY = "trainable_only"
+    EXCLUDED = "excluded"
+
+
+_TIER_BY_SPDX: dict[str, Tier] = {
+    "MIT": Tier.REDISTRIBUTABLE,
+    "Apache-2.0": Tier.REDISTRIBUTABLE,
+    "BSD-2-Clause": Tier.REDISTRIBUTABLE,
+    "BSD-3-Clause": Tier.REDISTRIBUTABLE,
+    "ISC": Tier.REDISTRIBUTABLE,
+    "Unlicense": Tier.REDISTRIBUTABLE,
+    "MPL-2.0": Tier.TRAINABLE_ONLY,
+    "LGPL-2.1": Tier.TRAINABLE_ONLY,
+    "LGPL-3.0": Tier.TRAINABLE_ONLY,
+    # GPL/AGPL and unknown are EXCLUDED: strong copyleft would bind the model
+    # outputs' redistribution story, and "unknown" defaults closed (V9).
+    "GPL-2.0": Tier.EXCLUDED,
+    "GPL-3.0": Tier.EXCLUDED,
+    "AGPL-3.0": Tier.EXCLUDED,
+}
+
+
+def license_tier(info: LicenseInfo) -> Tier:
+    """Map detected license → tier. Anything unrecognized is EXCLUDED — the
+    gate fails closed, never open (V9: the old substring filter failed open)."""
+    return _TIER_BY_SPDX.get(info.spdx_id, Tier.EXCLUDED)
+
+
+# ---------------------------------------------------------------------------
+# Benchmark decontamination (V3 / D-5)
+# ---------------------------------------------------------------------------
+
+#: The canonical 12 SWE-bench test repos (SWE-bench / -Lite / -Verified /
+#: -Multimodal all draw eval instances from these). Training on ANY of them
+#: contaminates every SWE-bench-family score we report (V3). Lowercase
+#: "org/repo" form. Extend via a JSON file (list of "org/repo" strings)
+#: passed to is_eval_contaminated(extra_list=...) — e.g. SWE-Gym eval splits.
+DECONTAMINATION_LIST: frozenset[str] = frozenset(
+    {
+        "astropy/astropy",
+        "django/django",
+        "matplotlib/matplotlib",
+        "mwaskom/seaborn",
+        "pallets/flask",
+        "psf/requests",
+        "pydata/xarray",
+        "pylint-dev/pylint",
+        "pytest-dev/pytest",
+        "scikit-learn/scikit-learn",
+        "sphinx-doc/sphinx",
+        "sympy/sympy",
+    }
+)
+
+
+def load_decontamination_list(path: Path) -> frozenset[str]:
+    """Load an extension list from a JSON file: ``["org/repo", ...]``. This is
+    THE documented mechanism for adding eval repos (new SWE-bench releases,
+    SWE-Gym eval splits) without editing code."""
+    entries = json.loads(path.read_text(encoding="utf-8"))
+    if not isinstance(entries, list):
+        raise ValueError(f"{path}: decontamination JSON must be a list of 'org/repo' strings")
+    return frozenset(normalize_repo(str(e)) for e in entries)
+
+
+def normalize_repo(repo: str) -> str:
+    """Reduce any repo spelling — full https/ssh GitHub URL, trailing ``.git``,
+    mixed case — to lowercase ``org/repo``. Decontamination must hit no matter
+    how the driver spells the repo (V3: a miss here is silent contamination)."""
+    r = repo.strip().lower()
+    r = re.sub(r"^(https?://|git@)", "", r)
+    r = re.sub(r"^[^/]*github\.com[:/]", "", r)
+    r = r.rstrip("/")
+    r = r.removesuffix(".git")
+    parts = [p for p in r.split("/") if p]
+    return "/".join(parts[:2]) if len(parts) >= 2 else r
+
+
+def is_eval_contaminated(repo: str, extra_list: frozenset[str] | None = None) -> bool:
+    """True if ``repo`` is in the SWE-bench-family eval set (or the caller's
+    extension list). Case-insensitive; accepts URLs and bare org/repo."""
+    key = normalize_repo(repo)
+    return key in DECONTAMINATION_LIST or (extra_list is not None and key in extra_list)
+
+
+# ---------------------------------------------------------------------------
+# The gate verdict — single entry point for the pipeline driver
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class GateVerdict:
+    """Everything the driver needs to admit/reject a repo, with reasons kept
+    for the run manifest (step 6's lineage record)."""
+
+    repo: str
+    license_info: LicenseInfo
+    tier: Tier
+    contaminated: bool
+    admitted: bool
+    reasons: list[str] = field(default_factory=list)
+
+
+def gate_repo(repo: str, repo_root: Path | None, extra_decontamination: frozenset[str] | None = None) -> GateVerdict:
+    """Architecture step 1: the one call the pipeline driver makes per repo.
+
+    Hard rules (in priority order):
+      1. Contaminated (V3) → NEVER admitted, even if the license is permissive.
+      2. Tier EXCLUDED (GPL/AGPL/unknown) → not admitted (V9: fail closed).
+      3. Tier TRAINABLE_ONLY → admitted, with the do-not-redistribute
+         constraint recorded as a reason so step 6 can route the rows.
+    """
+    contaminated = is_eval_contaminated(repo, extra_decontamination)
+    info = detect_license(repo_root) if repo_root is not None else LicenseInfo("unknown", "none")
+    tier = license_tier(info)
+
+    reasons: list[str] = []
+    if contaminated:
+        reasons.append(
+            f"benchmark decontamination: {normalize_repo(repo)} is a SWE-bench-family eval repo (V3/D-5)"
+        )
+    if repo_root is None:
+        reasons.append("no repo_root provided: license undetectable, failing closed (V9)")
+    if tier is Tier.EXCLUDED and not contaminated:
+        reasons.append(f"license tier EXCLUDED: spdx={info.spdx_id} (signal={info.signal})")
+    if tier is Tier.TRAINABLE_ONLY:
+        reasons.append(
+            f"license tier TRAINABLE_ONLY: spdx={info.spdx_id} — usable for training, "
+            "derivative diffs must NOT be redistributed (D-13)"
+        )
+
+    admitted = (not contaminated) and tier is not Tier.EXCLUDED
+    return GateVerdict(
+        repo=repo,
+        license_info=info,
+        tier=tier,
+        contaminated=contaminated,
+        admitted=admitted,
+        reasons=reasons,
+    )

composer_replication/datagen/rollout_harness.py

@@ -0,0 +1,214 @@
+"""rollout_harness.py — the agent loop over FeatureDeletionEnv (finding V2).
+
+THE critical missing component the design critic identified: nothing in the
+repo ran an agent episode against `FeatureDeletionEnv` to completion, so the
+SFT corpus had NO producer and the tree-of-work had no env-grounded seeds.
+`collect_trajectory` is that producer: prompt → policy.act → env.step → … →
+submit → `_grade()`, emitting a `CanonicalTrajectory` whose steps are real
+executed environment transitions (the seeds the tree needs, fixing the
+seed-trace/oracle disjointness of finding D-1 as a free byproduct).
+
+The policy is pluggable (`RolloutPolicy` protocol): a scripted fake for tests,
+a frontier API model for expert-trajectory collection (SWE-Gym/SWE-smith both
+validated this recipe — 491 and 5,016 expert trajectories respectively), or a
+local model later.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Protocol, runtime_checkable
+
+from composer_replication.datagen.env import FeatureDeletionEnv, StepResult
+from composer_replication.datagen.schema import FeatureDeletionTask
+from composer_replication.datagen.trajectory import (
+    CanonicalTrajectory,
+    ToolCall,
+    TrajectoryStep,
+)
+
+
+@runtime_checkable
+class RolloutPolicy(Protocol):
+    """Anything that maps (observation, history) → the next action.
+
+    Returning a `ToolCall` continues the episode (translated to an env action
+    dict); returning a plain `str` is the final message — the harness submits.
+    """
+
+    def act(self, observation: str, history: list[TrajectoryStep]) -> ToolCall | str: ...
+
+
+@dataclass
+class ScriptedPolicy:
+    """Test fake: replays a fixed action list, then submits."""
+
+    actions: list[ToolCall | str]
+    _i: int = 0
+
+    def act(self, observation: str, history: list[TrajectoryStep]) -> ToolCall | str:
+        if self._i >= len(self.actions):
+            return "done"  # str → submit
+        a = self.actions[self._i]
+        self._i += 1
+        return a
+
+
+class OpenRouterPolicy:
+    """Frontier-API policy for expert-trajectory collection (thin stub).
+
+    Mirrors `teacher_replay._call_teacher`'s payload shape (one chat call,
+    temperature 0.2). Lazy-deps on httpx so the module imports without it.
+    Deliberately minimal: real expert collection should evaluate adopting
+    mini-swe-agent/SWE-agent as the scaffold (deepread 11 finding 2) — this
+    class exists so the harness has a live-API path without a new framework.
+    """
+
+    def __init__(self, model_slug: str, api_key: str | None = None,
+                 max_tokens: int = 512) -> None:
+        try:
+            import httpx  # noqa: F401, PLC0415 — lazy heavy dep
+        except ImportError as e:
+            raise ImportError(
+                "OpenRouterPolicy requires httpx (`pip install httpx` or the "
+                "[serverless] extra). For tests use ScriptedPolicy. Got: " + repr(e)
+            ) from e
+        from composer_replication.teacher_replay import _load_api_key
+        self.model_slug = model_slug
+        self.api_key = api_key or _load_api_key()
+        self.max_tokens = max_tokens
+
+    def act(self, observation: str, history: list[TrajectoryStep]) -> ToolCall | str:
+        import httpx  # noqa: PLC0415
+
+        from composer_replication.teacher_replay import OPENROUTER_URL
+        messages = [{"role": "user", "content": observation}]
+        r = httpx.post(
+            OPENROUTER_URL,
+            json={"model": self.model_slug, "messages": messages,
+                  "max_tokens": self.max_tokens, "temperature": 0.2},
+            headers={"Authorization": f"Bearer {self.api_key}"},
+            timeout=120.0,
+        )
+        r.raise_for_status()
+        return str(r.json()["choices"][0]["message"]["content"])
+
+
+def _to_env_action(call: ToolCall) -> dict:
+    """ToolCall → FeatureDeletionEnv action dict.
+
+    CONVENTION (documented here, the single translation point): the env's
+    `step()` consumes ``{"type": <tool name>, **args}``; ``type=="submit"``
+    triggers grading (env.py:67). A ToolCall named "submit" therefore ends the
+    episode through the same path as a plain-text final message.
+    """
+    return {"type": call.name, **call.args}
+
+
+def collect_trajectory(
+    env: FeatureDeletionEnv,
+    task: FeatureDeletionTask,
+    policy: RolloutPolicy,
+    *,
+    max_turns: int = 40,
+    budget_usd: float | None = None,
+    provenance: dict | None = None,
+) -> CanonicalTrajectory:
+    """Run one episode and return the graded CanonicalTrajectory.
+
+    The episode ends when the policy emits a plain string (final message →
+    submit), a ToolCall named "submit", or `max_turns` is hit (the env grades
+    on its own turn limit too — we mirror it here so the harness's history
+    stays aligned with the env's accounting).
+    """
+    obs = env.reset(task)
+    steps: list[TrajectoryStep] = []
+    final: StepResult | None = None
+
+    for _ in range(max_turns):
+        action = policy.act(obs, steps)
+        if isinstance(action, str) or action.name == "submit":
+            final = env.step({"type": "submit"})
+            steps.append(TrajectoryStep(
+                observation=obs, action=action, result=final.observation,
+                tool_error=False,
+            ))
+            break
+        res = env.step(_to_env_action(action))
+        tool_error = "error" in (res.observation or "").lower()[:200]
+        steps.append(TrajectoryStep(
+            observation=obs, action=action, result=res.observation,
+            tool_error=tool_error,
+        ))
+        if res.done:  # env hit its own turn limit and graded
+            final = res
+            break
+        obs = res.observation
+
+    if final is None:
+        # max_turns exhausted without submit — grade what exists.
+        final = env.step({"type": "submit"})
+
+    info = final.info or {}
+    return CanonicalTrajectory(
+        task_id=task.task_id,
+        steps=steps,
+        grade=float(final.reward) if final.reward is not None else None,
+        guard_ok=bool(info.get("guard_ok", True)),
+        hacked=bool(info.get("hacked", False)),
+        provenance={"source": "rollout_harness",
+                    "policy": type(policy).__name__,
+                    **(provenance or {})},
+    )
+
+
+# ---------------------------------------------------------------------
+# Admission — type the signal and route it (final report §4)
+# ---------------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class AdmissionVerdict:
+    """Where a trajectory may go. Routing per the typed-train-on-all verdict:
+    clean full passes → SFT; clean near-misses → DPO-candidate (contrastive
+    rejected vs a winner, never raw negative gradient); everything else →
+    rejected (quarantine-side, full provenance kept for audit)."""
+
+    sft_admitted: bool
+    dpo_candidate: bool
+    rejected: bool
+    reasons: tuple[str, ...]
+
+
+def admit(traj: CanonicalTrajectory) -> AdmissionVerdict:
+    reasons: list[str] = []
+    clean = traj.guard_ok and not traj.hacked
+    if not traj.guard_ok:
+        reasons.append("pass_to_pass guard broken")
+    if traj.hacked:
+        reasons.append("hack monitor flagged")
+    grade = traj.grade if traj.grade is not None else 0.0
+    if traj.grade is None:
+        reasons.append("ungraded (no execution oracle)")
+
+    sft = clean and traj.grade is not None and grade == 1.0
+    dpo = clean and traj.grade is not None and 0.0 < grade < 1.0
+    if sft:
+        reasons.append("clean full pass")
+    elif dpo:
+        reasons.append(f"clean near-miss (grade={grade:.2f})")
+    elif clean and grade == 0.0 and traj.grade is not None:
+        reasons.append("clean zero — no partial signal")
+    return AdmissionVerdict(
+        sft_admitted=sft, dpo_candidate=dpo,
+        rejected=not (sft or dpo), reasons=tuple(reasons),
+    )
+
+
+__all__ = [
+    "RolloutPolicy",
+    "ScriptedPolicy",
+    "OpenRouterPolicy",
+    "collect_trajectory",
+    "AdmissionVerdict",
+    "admit",
+]

composer_replication/datagen/swesmith_adapter.py

@@ -0,0 +1,269 @@
+"""swesmith_adapter.py — adapt SWE-smith instances into Feature-Deletion tasks.
+
+THE BUY-VS-BUILD VERDICT (deepread finding V4 / D-6): `pip install swesmith`
+(MIT) already ships what ADR-010's "Option B greenfield generator" would have
+hand-built — env construction from arbitrary GitHub repos (ONE Docker image per
+repo, ~500x more storage-efficient than per-task images), five bug-synthesis
+strategies, issue-text generation, and validation-by-test-execution, at a
+verified $1,360 + ~20 human-hours for 50k tasks. Its **PR Mirror strategy is
+exactly this repo's gold-patch-reversion mechanic** and SWE-smith's own ablation
+(Table 5, arXiv:2504.21798) shows PR-Mirror trajectories train the BEST models
+of its five strategies — independent validation of ADR-010's core approach.
+So SWE-smith is the synthesis ENGINE for "point at a repo"; this module is the
+schema bridge into the existing `FeatureDeletionTask` world.
+
+THE SEMANTIC INVERSION (load-bearing — easy to get backwards):
+  * SWE-bench-shaped instances: `patch` is the GOLD FIX. broken = HEAD with the
+    fix reverted (`git apply -R patch`). `SweBenchAdapter` stores `patch` as
+    `golden_diff` directly.
+  * SWE-smith instances: `patch` INTRODUCES THE BUG. broken = HEAD with the
+    patch APPLIED. The fix — what the agent must produce, the validator's gate-4
+    restoration diff — is the REVERSE of the bug patch.
+This adapter therefore stores `golden_diff = reverse_unified_diff(bug_patch)`.
+When mechanical reversal fails (exotic diff features), it falls back to the
+original patch tagged with a provenance marker so downstream gate-4 validation
+knows to use `git apply -R` instead of `git apply`.
+
+The adapter itself needs nothing beyond core deps. Live synthesis (building new
+repo profiles / generating new bugs) needs the `swesmith` toolkit + Docker on
+Linux — see the `[swesmith]` extra in pyproject.
+"""
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass
+
+from composer_replication.datagen.schema import FeatureDeletionTask
+from composer_replication.datagen.substrates import _as_tuple
+
+#: Marker prefixed to golden_diff when reverse_unified_diff could not invert the
+#: bug patch mechanically. Consumers (gate-4 validation) must then apply the
+#: remainder with `git apply -R` (it is the FORWARD bug patch, not the fix).
+UNREVERSED_MARKER = "### UNREVERSED-BUG-PATCH (apply with -R) ###\n"
+
+#: instance_id substring patterns → synthesis strategy (SWE-smith §2.1 / §B).
+#: Patterns follow the toolkit's naming: e.g.
+#:   pandas-dev__pandas.abc123.lm_modify__xyz
+#:   ...func_pm_ctrl_invert_if__..., ...combine_file__..., ...pr_1234
+_STRATEGY_PATTERNS: tuple[tuple[str, str], ...] = (
+    ("lm_modify", "lm_modify"),
+    ("lm_rewrite", "lm_rewrite"),
+    ("func_pm", "procedural"),     # procedural AST modifications (13 transform types)
+    ("func_basic", "procedural"),
+    ("combine_file", "combine"),
+    ("combine_module", "combine"),
+    ("combine", "combine"),
+    ("pr_", "pr_mirror"),
+)
+
+
+def parse_strategy(instance_id: str) -> str:
+    """Map a SWE-smith instance_id to its bug-synthesis strategy.
+
+    Returns one of {lm_modify, lm_rewrite, procedural, combine, pr_mirror,
+    unknown}. The strategy matters because SWE-smith's Table 5 ablation found
+    trajectory quality differs sharply by strategy (PR Mirror best, LM Modify
+    steep drop-off) — we carry it as provenance so corpus builds can weight or
+    filter by strategy.
+    """
+    iid = (instance_id or "").lower()
+    for pattern, strategy in _STRATEGY_PATTERNS:
+        if pattern in iid:
+            return strategy
+    return "unknown"
+
+
+#: Heuristic cold-start difficulty priors per strategy, motivated by SWE-smith
+#: Table 1 medians (PR Mirror: 3 median F2P but 14 lines edited; Combine: 15
+#: F2P / 11 lines = multi-site; procedural: 7 F2P / 5 lines, mechanical).
+#: These only seed DifficultyCurriculum's p-hat before real rollouts exist.
+_DIFFICULTY_PRIOR: dict[str, float] = {
+    "pr_mirror": 0.4,
+    "combine": 0.4,
+    "lm_rewrite": 0.45,
+    "lm_modify": 0.55,
+    "procedural": 0.6,
+    "unknown": 0.5,
+}
+
+
+_HUNK_RE = re.compile(
+    r"^@@ -(?P<old_start>\d+)(?:,(?P<old_count>\d+))? "
+    r"\+(?P<new_start>\d+)(?:,(?P<new_count>\d+))? @@(?P<tail>.*)$"
+)
+
+
+def reverse_unified_diff(patch: str) -> str | None:
+    """Mechanically invert a unified diff (swap additions and deletions).
+
+    Handles the standard unified-diff features SWE-smith patches use:
+    ``diff --git`` headers, ``---``/``+++`` file lines, ``@@`` hunk headers
+    (old/new ranges swapped), ``+``/``-`` body lines (swapped), context lines,
+    and ``\`` markers (kept in place).
+
+    HONEST LIMITATIONS (returns None — caller falls back to UNREVERSED_MARKER):
+      * file mode changes (``old mode``/``new mode``), renames/copies
+        (``rename from``...), binary patches (``GIT binary patch``), and
+        ``index`` lines with mode suffixes are NOT inverted — reversing them
+        correctly requires git plumbing, not text surgery.
+      * Within a hunk, a reversed diff's line ORDER for paired -/+ runs is the
+        naive swap; `git apply` accepts it, but it is not byte-identical to
+        what `git diff` would emit for the reverse change.
+    """
+    if not patch or "@@" not in patch:
+        return None
+    unsupported = ("old mode ", "new mode ", "rename from ", "rename to ",
+                   "copy from ", "copy to ", "GIT binary patch")
+    if any(marker in patch for marker in unsupported):
+        return None
+
+    out: list[str] = []
+    for line in patch.splitlines():
+        if line.startswith("diff --git "):
+            # `diff --git a/<old> b/<new>` → swap the two paths.
+            m = re.match(r"^diff --git a/(?P<a>.+) b/(?P<b>.+)$", line)
+            if m:
+                out.append(f"diff --git a/{m.group('b')} b/{m.group('a')}")
+            else:
+                out.append(line)
+        elif line.startswith("--- "):
+            out.append("+++ " + line[4:].replace("a/", "b/", 1)
+                       if line[4:].startswith("a/") else "+++ " + line[4:])
+        elif line.startswith("+++ "):
+            out.append("--- " + line[4:].replace("b/", "a/", 1)
+                       if line[4:].startswith("b/") else "--- " + line[4:])
+        elif line.startswith("@@"):
+            m = _HUNK_RE.match(line)
+            if not m:
+                return None
+            old_start, old_count = m.group("old_start"), m.group("old_count")
+            new_start, new_count = m.group("new_start"), m.group("new_count")
+            oc = f",{old_count}" if old_count is not None else ""
+            nc = f",{new_count}" if new_count is not None else ""
+            out.append(f"@@ -{new_start}{nc} +{old_start}{oc} @@{m.group('tail')}")
+        elif line.startswith("+"):
+            out.append("-" + line[1:])
+        elif line.startswith("-"):
+            out.append("+" + line[1:])
+        else:
+            # context lines, `index ...`, `\ No newline...` pass through.
+            out.append(line)
+    return "\n".join(out) + ("\n" if patch.endswith("\n") else "")
+
+
+@dataclass(frozen=True)
+class SwesmithMeta:
+    """Sidecar provenance for a SWE-smith-derived task.
+
+    Kept OUT of the frozen `FeatureDeletionTask` schema deliberately — the
+    schema is shared with SweBenchAdapter and the trainer; strategy provenance
+    is a corpus-construction concern, carried alongside (e.g. into the run
+    manifest), never into the policy-visible task row.
+    """
+
+    strategy: str            # lm_modify | lm_rewrite | procedural | combine | pr_mirror | unknown
+    diff_reversed: bool      # True if golden_diff is the mechanical reverse of the bug patch
+    source: str = "swesmith"
+
+
+@dataclass
+class SwesmithAdapter:
+    """Convert a SWE-smith instance dict into a FeatureDeletionTask.
+
+    Mirrors `SweBenchAdapter`'s shape; differs in the patch semantics (see the
+    module docstring INVERSION note) and the per-REPO image convention.
+    """
+
+    default_test_command: str = "python -m pytest -q"
+
+    def image_for(self, instance: dict) -> str:
+        # SWE-smith publishes ONE image per repo (not per task). Rows carry
+        # `image_name`; some exports use `docker_image`. Fall back to the
+        # toolkit's naming convention derived from the repo slug.
+        for key in ("image_name", "docker_image"):
+            if instance.get(key):
+                return str(instance[key])
+        repo = str(instance.get("repo", "unknown")).replace("/", "__").lower()
+        return f"swesmith.x86_64.{repo}:latest"
+
+    def to_task(self, instance: dict) -> FeatureDeletionTask:
+        task, _meta = self.to_task_with_meta(instance)
+        return task
+
+    def to_task_with_meta(self, instance: dict) -> tuple[FeatureDeletionTask, SwesmithMeta]:
+        iid = str(instance.get("instance_id") or instance.get("task_id") or "unknown")
+        strategy = parse_strategy(iid)
+
+        bug_patch = str(instance.get("patch", ""))
+        fix = reverse_unified_diff(bug_patch)
+        if fix is not None:
+            golden_diff = fix
+            diff_reversed = True
+        else:
+            golden_diff = UNREVERSED_MARKER + bug_patch
+            diff_reversed = False
+
+        ftp = _as_tuple(instance.get("FAIL_TO_PASS"))
+        ptp = _as_tuple(instance.get("PASS_TO_PASS"))
+
+        task = FeatureDeletionTask(
+            task_id=iid,
+            repo=str(instance.get("repo", "unknown")),
+            base_commit=str(instance.get("base_commit", "")),
+            broken_image=self.image_for(instance),
+            test_command=str(instance.get("test_command") or self.default_test_command),
+            fail_to_pass=ftp,
+            pass_to_pass=ptp,
+            golden_diff=golden_diff,
+            granularity="feature",
+            # SWE-smith rows don't carry per-instance licenses; repo-level
+            # licensing is the repo_gate's job (deepread finding V9/D-13).
+            upstream_license=str(instance.get("license_name", "unknown")),
+            difficulty_prior=_DIFFICULTY_PRIOR.get(strategy, 0.5),
+        )
+        return task, SwesmithMeta(strategy=strategy, diff_reversed=diff_reversed)
+
+
+def load_swesmith_instances(
+    path_or_hf_id: str,
+    *,
+    limit: int | None = None,
+) -> list[dict]:
+    """Load SWE-smith instances from a local JSONL file or the HF dataset.
+
+    Local ``.jsonl`` paths need no extra deps (used by tests/fixtures). HF ids
+    (e.g. ``SWE-bench/SWE-smith``) lazy-import `datasets` from the `[datagen]`
+    extra.
+    """
+    if path_or_hf_id.endswith(".jsonl"):
+        rows: list[dict] = []
+        with open(path_or_hf_id, encoding="utf-8") as f:
+            for line in f:
+                line = line.strip()
+                if not line:
+                    continue
+                rows.append(json.loads(line))
+                if limit is not None and len(rows) >= limit:
+                    break
+        return rows
+    try:
+        from datasets import load_dataset  # noqa: PLC0415 — lazy heavy dep
+    except ImportError as e:
+        raise RuntimeError(
+            "Loading SWE-smith from the HF Hub requires `datasets`; install "
+            "with `pip install -e .[datagen]`. Got: " + repr(e)
+        ) from e
+    split = load_dataset(path_or_hf_id, split="train")
+    rows = [dict(r) for i, r in enumerate(split) if limit is None or i < limit]
+    return rows[: limit if limit is not None else len(rows)]
+
+
+__all__ = [
+    "SwesmithAdapter",
+    "SwesmithMeta",
+    "UNREVERSED_MARKER",
+    "load_swesmith_instances",
+    "parse_strategy",
+    "reverse_unified_diff",
+]

composer_replication/datagen/tests/test_repo_gate.py

@@ -0,0 +1,419 @@
+"""Tests for the Stage-0 ingest gate (repo_gate.py) — architecture step 1.
+
+Coverage targets the two findings the module closes:
+  * V9/D-13 — SPDX detection from real license fixture texts (incl. the
+    tricky GNU-family cross-citation and Apache-vs-MIT phrasing) and the
+    three-tier mapping that replaces the old boolean substring filter.
+  * V3/D-5  — decontamination hits in exact, URL, and mixed-case forms, and
+    the hard never-admit rule in the composed verdict.
+
+CPU-only, stdlib + tmp_path fixtures — no network, no Docker.
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from composer_replication.datagen.repo_gate import (
+    DECONTAMINATION_LIST,
+    GateVerdict,
+    LicenseInfo,
+    Tier,
+    detect_license,
+    gate_repo,
+    is_eval_contaminated,
+    license_tier,
+    load_decontamination_list,
+    normalize_repo,
+)
+
+# ---------------------------------------------------------------------
+# License fixture texts — distinctive excerpts of the real license texts.
+# ---------------------------------------------------------------------
+
+MIT_TEXT = """\
+MIT License
+
+Copyright (c) 2026 Example Org
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction...
+"""
+
+# The tricky Apache case: the words "permission" and "license" appear in both
+# MIT and Apache; only Apache names itself with a version.
+APACHE_TEXT = """\
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+   "License" shall mean the terms and conditions for use, reproduction,
+   and distribution as defined by Sections 1 through 9 of this document.
+"""
+
+BSD3_TEXT = """\
+BSD 3-Clause License
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice...
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+"""
+
+BSD2_TEXT = """\
+BSD 2-Clause License
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice.
+2. Redistributions in binary form must reproduce the above copyright notice.
+"""
+
+ISC_TEXT = """\
+ISC License
+
+Copyright (c) 2026, Example Org
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+"""
+
+# GPL-3.0 §13 cross-cites the AGPL by full name — the classic trap for
+# full-body substring matchers. Header anchoring must win.
+GPL3_TEXT = """\
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc.
+
+  13. Use with the GNU Affero General Public License.
+  Notwithstanding any other provision of this License, you have
+  permission to link or combine any covered work with a work licensed
+  under version 3 of the GNU Affero General Public License...
+"""
+
+GPL2_TEXT = """\
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+"""
+
+# AGPL-3.0 §13 reciprocally cites the plain GPL by name.
+AGPL3_TEXT = """\
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+  Notwithstanding any other provision of this License...
+"""
+
+LGPL21_TEXT = """\
+                  GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 2.1, February 1999
+
+ Copyright (C) 1991, 1999 Free Software Foundation, Inc.
+"""
+
+MPL2_TEXT = """\
+Mozilla Public License Version 2.0
+==================================
+
+1. Definitions
+--------------
+1.1. "Contributor"
+    means each individual or legal entity that creates, contributes to
+    the creation of, or owns Covered Software.
+"""
+
+UNLICENSE_TEXT = """\
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any means.
+"""
+
+
+def _repo_with_license(tmp_path: Path, text: str, filename: str = "LICENSE") -> Path:
+    (tmp_path / filename).write_text(text, encoding="utf-8")
+    return tmp_path
+
+
+# ---------------------------------------------------------------------
+# detect_license — SPDX classification from file text
+# ---------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    ("text", "expected"),
+    [
+        (MIT_TEXT, "MIT"),
+        (APACHE_TEXT, "Apache-2.0"),
+        (BSD3_TEXT, "BSD-3-Clause"),
+        (BSD2_TEXT, "BSD-2-Clause"),
+        (ISC_TEXT, "ISC"),
+        (GPL3_TEXT, "GPL-3.0"),
+        (GPL2_TEXT, "GPL-2.0"),
+        (AGPL3_TEXT, "AGPL-3.0"),
+        (LGPL21_TEXT, "LGPL-2.1"),
+        (MPL2_TEXT, "MPL-2.0"),
+        (UNLICENSE_TEXT, "Unlicense"),
+    ],
+    ids=["mit", "apache2", "bsd3", "bsd2", "isc", "gpl3", "gpl2", "agpl3", "lgpl21", "mpl2", "unlicense"],
+)
+def test_detect_license_spdx_ids(tmp_path: Path, text: str, expected: str):
+    info = detect_license(_repo_with_license(tmp_path, text))
+    assert info.spdx_id == expected
+    assert info.signal == "license_file"
+    assert info.source == "LICENSE"
+
+
+def test_gpl3_not_misread_as_agpl(tmp_path: Path):
+    """GPL-3.0 §13 names the AGPL in its body; header anchoring must keep
+    this classified as GPL-3.0 (the V9 substring filter would have tripped)."""
+    info = detect_license(_repo_with_license(tmp_path, GPL3_TEXT))
+    assert info.spdx_id == "GPL-3.0"
+
+
+def test_apache_notice_without_header_still_apache(tmp_path: Path):
+    """The short 'Licensed under the Apache License, Version 2.0' boilerplate
+    has no canonical header — the body fallback must catch it, and must not
+    fall through to MIT despite shared 'permission' vocabulary."""
+    notice = (
+        "Copyright 2026 Example Org\n\n"
+        "Licensed under the Apache License, Version 2.0 (the \"License\");\n"
+        "you may not use this file except in compliance with the License.\n"
+    )
+    info = detect_license(_repo_with_license(tmp_path, notice))
+    assert info.spdx_id == "Apache-2.0"
+
+
+def test_detect_license_alternate_filenames(tmp_path: Path):
+    info = detect_license(_repo_with_license(tmp_path, GPL2_TEXT, filename="COPYING"))
+    assert info.spdx_id == "GPL-2.0"
+    assert info.source == "COPYING"
+    info2 = detect_license(_repo_with_license(tmp_path, MIT_TEXT, filename="LICENSE.md"))
+    # LICENSE.md is also present in tmp_path now alongside COPYING; first
+    # filename in priority order (LICENSE/LICENSE.txt/LICENSE.md) wins over COPYING.
+    assert info2.spdx_id == "MIT"
+    assert info2.source == "LICENSE.md"
+
+
+def test_detect_license_unknown_text(tmp_path: Path):
+    info = detect_license(_repo_with_license(tmp_path, "All rights reserved. Ask legal."))
+    assert info.spdx_id == "unknown"
+
+
+def test_detect_license_no_files(tmp_path: Path):
+    info = detect_license(tmp_path)
+    assert info == LicenseInfo(spdx_id="unknown", signal="none")
+
+
+def test_classifier_secondary_signal(tmp_path: Path):
+    """No LICENSE file, but pyproject carries a trove classifier — the
+    classifier signal must win and be recorded as such."""
+    (tmp_path / "pyproject.toml").write_text(
+        '[project]\nname = "x"\nclassifiers = [\n'
+        '    "License :: OSI Approved :: MIT License",\n]\n',
+        encoding="utf-8",
+    )
+    info = detect_license(tmp_path)
+    assert info.spdx_id == "MIT"
+    assert info.signal == "classifier"
+    assert info.source == "pyproject.toml"
+
+
+def test_classifier_pep639_expression(tmp_path: Path):
+    (tmp_path / "pyproject.toml").write_text(
+        '[project]\nname = "x"\nlicense = "Apache-2.0"\n', encoding="utf-8"
+    )
+    info = detect_license(tmp_path)
+    assert info.spdx_id == "Apache-2.0"
+    assert info.signal == "classifier"
+
+
+def test_license_file_beats_classifier(tmp_path: Path):
+    """When both signals exist and the file is classifiable, the file wins —
+    the classifier is secondary by design (it can't tell BSD-2 from BSD-3)."""
+    _repo_with_license(tmp_path, GPL3_TEXT)
+    (tmp_path / "pyproject.toml").write_text(
+        'classifiers = ["License :: OSI Approved :: MIT License"]\n', encoding="utf-8"
+    )
+    info = detect_license(tmp_path)
+    assert info.spdx_id == "GPL-3.0"
+    assert info.signal == "license_file"
+
+
+def test_unclassifiable_file_falls_back_to_classifier(tmp_path: Path):
+    _repo_with_license(tmp_path, "Custom corporate license, see legal dept.")
+    (tmp_path / "pyproject.toml").write_text(
+        'classifiers = ["License :: OSI Approved :: ISC License"]\n', encoding="utf-8"
+    )
+    info = detect_license(tmp_path)
+    assert info.spdx_id == "ISC"
+    assert info.signal == "classifier"
+
+
+# ---------------------------------------------------------------------
+# license_tier — tiers, not a boolean (D-13)
+# ---------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    ("spdx", "tier"),
+    [
+        ("MIT", Tier.REDISTRIBUTABLE),
+        ("Apache-2.0", Tier.REDISTRIBUTABLE),
+        ("BSD-2-Clause", Tier.REDISTRIBUTABLE),
+        ("BSD-3-Clause", Tier.REDISTRIBUTABLE),
+        ("ISC", Tier.REDISTRIBUTABLE),
+        ("Unlicense", Tier.REDISTRIBUTABLE),
+        ("MPL-2.0", Tier.TRAINABLE_ONLY),
+        ("LGPL-2.1", Tier.TRAINABLE_ONLY),
+        ("LGPL-3.0", Tier.TRAINABLE_ONLY),
+        ("GPL-2.0", Tier.EXCLUDED),
+        ("GPL-3.0", Tier.EXCLUDED),
+        ("AGPL-3.0", Tier.EXCLUDED),
+        ("unknown", Tier.EXCLUDED),
+        ("WTFPL", Tier.EXCLUDED),  # unrecognized id → fail closed
+    ],
+)
+def test_license_tier_mapping(spdx: str, tier: Tier):
+    assert license_tier(LicenseInfo(spdx_id=spdx, signal="license_file")) is tier
+
+
+# ---------------------------------------------------------------------
+# Decontamination (V3 / D-5)
+# ---------------------------------------------------------------------
+
+
+def test_decontamination_list_has_the_canonical_12():
+    assert len(DECONTAMINATION_LIST) == 12
+    assert "django/django" in DECONTAMINATION_LIST
+    assert "sympy/sympy" in DECONTAMINATION_LIST
+
+
+@pytest.mark.parametrize(
+    "repo",
+    [
+        "django/django",  # exact
+        "Django/Django",  # case
+        "https://github.com/django/django",  # https URL
+        "https://github.com/django/django.git",  # URL + .git
+        "git@github.com:django/django.git",  # ssh URL
+        "https://github.com/django/django/",  # trailing slash
+    ],
+)
+def test_is_eval_contaminated_hits(repo: str):
+    assert is_eval_contaminated(repo) is True
+
+
+@pytest.mark.parametrize(
+    "repo",
+    [
+        "pandas-dev/pandas",
+        "https://github.com/torvalds/linux",
+        "someuser/django",  # fork-org differs: NOT the eval repo
+    ],
+)
+def test_is_eval_contaminated_misses(repo: str):
+    assert is_eval_contaminated(repo) is False
+
+
+def test_normalize_repo_forms():
+    assert normalize_repo("git@github.com:PSF/Requests.git") == "psf/requests"
+    assert normalize_repo("https://github.com/pydata/xarray/tree/main") == "pydata/xarray"
+
+
+def test_extension_list_from_json(tmp_path: Path):
+    """The documented extension mechanism: extra eval repos load from JSON
+    and hit through the same normalized matching."""
+    extra_path = tmp_path / "extra.json"
+    extra_path.write_text(json.dumps(["SWE-Gym/Extra-Repo"]), encoding="utf-8")
+    extra = load_decontamination_list(extra_path)
+    assert is_eval_contaminated("https://github.com/swe-gym/extra-repo", extra_list=extra)
+    assert not is_eval_contaminated("swe-gym/other-repo", extra_list=extra)
+
+
+def test_extension_list_rejects_non_list(tmp_path: Path):
+    bad = tmp_path / "bad.json"
+    bad.write_text('{"repo": "a/b"}', encoding="utf-8")
+    with pytest.raises(ValueError):
+        load_decontamination_list(bad)
+
+
+# ---------------------------------------------------------------------
+# gate_repo — verdict composition
+# ---------------------------------------------------------------------
+
+
+def test_gate_admits_permissive_clean_repo(tmp_path: Path):
+    v = gate_repo("example/clean", _repo_with_license(tmp_path, MIT_TEXT))
+    assert isinstance(v, GateVerdict)
+    assert v.admitted is True
+    assert v.tier is Tier.REDISTRIBUTABLE
+    assert v.contaminated is False
+    assert v.reasons == []
+
+
+def test_gate_contaminated_never_admitted_even_if_permissive(tmp_path: Path):
+    """V3 hard rule: an eval repo with an MIT license is STILL rejected —
+    decontamination outranks license."""
+    v = gate_repo("https://github.com/pallets/flask", _repo_with_license(tmp_path, MIT_TEXT))
+    assert v.contaminated is True
+    assert v.admitted is False
+    assert any("decontamination" in r for r in v.reasons)
+    # license detection still ran and is recorded for the manifest
+    assert v.license_info.spdx_id == "MIT"
+
+
+def test_gate_excluded_tier_never_admitted(tmp_path: Path):
+    v = gate_repo("example/agpl-repo", _repo_with_license(tmp_path, AGPL3_TEXT))
+    assert v.tier is Tier.EXCLUDED
+    assert v.admitted is False
+    assert any("EXCLUDED" in r for r in v.reasons)
+
+
+def test_gate_trainable_only_admitted_with_reason(tmp_path: Path):
+    """D-13: weak copyleft is admitted for training, but the verdict must
+    carry the do-not-redistribute constraint for step 6 to route on."""
+    v = gate_repo("example/mpl-repo", _repo_with_license(tmp_path, MPL2_TEXT))
+    assert v.tier is Tier.TRAINABLE_ONLY
+    assert v.admitted is True
+    assert any("TRAINABLE_ONLY" in r for r in v.reasons)
+    assert any("redistributed" in r for r in v.reasons)
+
+
+def test_gate_no_repo_root_fails_closed():
+    """No repo_root → license undetectable → unknown → EXCLUDED → rejected
+    (V9: the gate must default closed, never open)."""
+    v = gate_repo("example/unfetched", None)
+    assert v.license_info.spdx_id == "unknown"
+    assert v.tier is Tier.EXCLUDED
+    assert v.admitted is False
+    assert any("failing closed" in r for r in v.reasons)
+
+
+def test_gate_extra_decontamination_list(tmp_path: Path):
+    extra = frozenset({"my-eval/secret-benchmark"})
+    v = gate_repo(
+        "https://github.com/My-Eval/Secret-Benchmark.git",
+        _repo_with_license(tmp_path, MIT_TEXT),
+        extra_decontamination=extra,
+    )
+    assert v.contaminated is True
+    assert v.admitted is False

composer_replication/datagen/tests/test_rollout_harness.py

@@ -0,0 +1,103 @@
+"""Tests for the rollout harness (deepread finding V2 — the SFT-corpus producer)."""
+from __future__ import annotations
+
+from composer_replication.datagen.env import FeatureDeletionEnv
+from composer_replication.datagen.rollout_harness import (
+    ScriptedPolicy,
+    admit,
+    collect_trajectory,
+)
+from composer_replication.datagen.sandbox import FakeSandbox
+from composer_replication.datagen.schema import FeatureDeletionTask
+from composer_replication.datagen.trajectory import CanonicalTrajectory, ToolCall
+
+
+def _task() -> FeatureDeletionTask:
+    return FeatureDeletionTask(
+        task_id="t1", repo="org/repo", base_commit="abc",
+        broken_image="img:1", test_command="pytest -q",
+        fail_to_pass=("t/a.py::t1", "t/a.py::t2"),
+        pass_to_pass=("t/a.py::keep",),
+    )
+
+
+def _env(outcomes: dict[str, bool]) -> FeatureDeletionEnv:
+    return FeatureDeletionEnv(FakeSandbox(test_outcomes=outcomes))
+
+
+def test_collect_trajectory_full_pass():
+    """Policy 'fixes' the repo via the FakeSandbox set_outcome pseudo-action,
+    then submits — grade 1.0, steps record real env transitions."""
+    env = _env({"t/a.py::keep": True})
+    policy = ScriptedPolicy(actions=[
+        ToolCall("set_outcome", {"outcomes": {"t/a.py::t1": True, "t/a.py::t2": True}}),
+        "final answer: implemented the feature",
+    ])
+    traj = collect_trajectory(env, _task(), policy)
+    assert isinstance(traj, CanonicalTrajectory)
+    assert traj.grade == 1.0
+    assert traj.guard_ok is True and traj.hacked is False
+    assert len(traj.steps) == 2
+    assert isinstance(traj.steps[0].action, ToolCall)
+    assert traj.steps[0].result == "ok"           # env.step observation recorded
+    assert traj.provenance["source"] == "rollout_harness"
+
+
+def test_collect_trajectory_guard_broken_zeroes_reward():
+    env = _env({"t/a.py::keep": False})  # functional guard broken
+    policy = ScriptedPolicy(actions=[
+        ToolCall("set_outcome", {"outcomes": {"t/a.py::t1": True, "t/a.py::t2": True,
+                                              "t/a.py::keep": False}}),
+        "done",
+    ])
+    traj = collect_trajectory(env, _task(), policy)
+    assert traj.grade == 0.0
+    assert traj.guard_ok is False
+
+
+def test_collect_trajectory_near_miss():
+    env = _env({"t/a.py::keep": True})
+    policy = ScriptedPolicy(actions=[
+        ToolCall("set_outcome", {"outcomes": {"t/a.py::t1": True}}),  # 1 of 2
+        "done",
+    ])
+    traj = collect_trajectory(env, _task(), policy)
+    assert traj.grade == 0.5
+    assert traj.guard_ok is True
+
+
+def test_collect_trajectory_max_turns_grades_anyway():
+    env = _env({"t/a.py::keep": True})
+    looping = ScriptedPolicy(actions=[ToolCall("bash", {"command": "ls"})] * 50)
+    traj = collect_trajectory(env, _task(), looping, max_turns=3)
+    assert traj.grade is not None  # graded despite never submitting
+
+
+# ---------------------------------------------------------------------
+# Admission routing (typed train-on-all, final report §4)
+# ---------------------------------------------------------------------
+
+
+def _t(grade, guard_ok=True, hacked=False) -> CanonicalTrajectory:
+    return CanonicalTrajectory(task_id="x", grade=grade, guard_ok=guard_ok, hacked=hacked)
+
+
+def test_admit_routes_clean_pass_to_sft():
+    v = admit(_t(1.0))
+    assert v.sft_admitted and not v.dpo_candidate and not v.rejected
+
+
+def test_admit_routes_near_miss_to_dpo():
+    v = admit(_t(0.5))
+    assert v.dpo_candidate and not v.sft_admitted and not v.rejected
+
+
+def test_admit_rejects_hacked_even_at_full_grade():
+    v = admit(_t(1.0, hacked=True))
+    assert v.rejected and "hack monitor flagged" in v.reasons
+
+
+def test_admit_rejects_guard_broken_and_ungraded():
+    assert admit(_t(1.0, guard_ok=False)).rejected
+    assert admit(_t(None)).rejected
+    assert admit(_t(0.0)).rejected

composer_replication/datagen/tests/test_swesmith_adapter.py

@@ -0,0 +1,165 @@
+"""Tests for the SWE-smith adapter (deepread finding V4 — buy-vs-build).
+
+The load-bearing coverage: the PATCH-SEMANTICS INVERSION (SWE-smith's patch
+introduces the bug; golden_diff must be its reverse) and the mechanical
+reverse_unified_diff round-trip.
+"""
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from composer_replication.datagen.schema import FeatureDeletionTask
+from composer_replication.datagen.substrates import SweBenchAdapter
+from composer_replication.datagen.swesmith_adapter import (
+    UNREVERSED_MARKER,
+    SwesmithAdapter,
+    load_swesmith_instances,
+    parse_strategy,
+    reverse_unified_diff,
+)
+
+BUG_PATCH = """\
+diff --git a/pkg/mod.py b/pkg/mod.py
+index 1111111..2222222 100644
+--- a/pkg/mod.py
++++ b/pkg/mod.py
+@@ -1,4 +1,3 @@
+ def add(a, b):
+-    return a + b
++    return a - b
+ # trailing context
+"""
+
+
+def _instance(**over) -> dict:
+    base = {
+        "instance_id": "getmoto__moto.abc1234.lm_modify__1a2b",
+        "repo": "getmoto/moto",
+        "base_commit": "abc1234",
+        "patch": BUG_PATCH,
+        "FAIL_TO_PASS": json.dumps(["tests/test_mod.py::test_add"]),
+        "PASS_TO_PASS": json.dumps(["tests/test_mod.py::test_other"]),
+        "image_name": "swesmith.x86_64.getmoto__moto:latest",
+    }
+    base.update(over)
+    return base
+
+
+# ---------------------------------------------------------------------
+# Strategy parsing
+# ---------------------------------------------------------------------
+
+
+@pytest.mark.parametrize("iid,expected", [
+    ("r__x.abc.lm_modify__1", "lm_modify"),
+    ("r__x.abc.lm_rewrite__1", "lm_rewrite"),
+    ("r__x.abc.func_pm_ctrl_invert_if__1", "procedural"),
+    ("r__x.abc.func_basic__1", "procedural"),
+    ("r__x.abc.combine_file__1", "combine"),
+    ("r__x.abc.combine_module__2", "combine"),
+    ("r__x.abc.pr_1234", "pr_mirror"),
+    ("r__x.abc.mystery__1", "unknown"),
+])
+def test_parse_strategy(iid, expected):
+    assert parse_strategy(iid) == expected
+
+
+# ---------------------------------------------------------------------
+# reverse_unified_diff
+# ---------------------------------------------------------------------
+
+
+def test_reverse_swaps_adds_and_removes():
+    rev = reverse_unified_diff(BUG_PATCH)
+    assert rev is not None
+    # The bug ADDED "return a - b"; the reverse must REMOVE it.
+    assert "-    return a - b" in rev
+    assert "+    return a + b" in rev
+    # Hunk header ranges swapped: -1,4 +1,3 → -1,3 +1,4
+    assert "@@ -1,3 +1,4 @@" in rev
+    # Context lines untouched.
+    assert " def add(a, b):" in rev
+    assert " # trailing context" in rev
+
+
+def test_reverse_round_trip_is_identity_on_body():
+    rev = reverse_unified_diff(BUG_PATCH)
+    rev2 = reverse_unified_diff(rev)
+    # Round trip restores hunks and +/- bodies (headers may normalize).
+    orig_body = [ln for ln in BUG_PATCH.splitlines() if ln[:1] in "+-@" and not ln.startswith(("+++", "---"))]
+    rt_body = [ln for ln in rev2.splitlines() if ln[:1] in "+-@" and not ln.startswith(("+++", "---"))]
+    assert orig_body == rt_body
+
+
+def test_reverse_refuses_renames_and_binary():
+    assert reverse_unified_diff("diff --git a/x b/y\nrename from x\nrename to y\n") is None
+    assert reverse_unified_diff("diff --git a/x b/x\nGIT binary patch\nliteral 5\n") is None
+    assert reverse_unified_diff("") is None
+    assert reverse_unified_diff("no hunks here") is None
+
+
+# ---------------------------------------------------------------------
+# Adapter
+# ---------------------------------------------------------------------
+
+
+def test_to_task_golden_diff_is_the_fix_not_the_bug():
+    """THE semantic inversion: golden_diff must restore the feature."""
+    task, meta = SwesmithAdapter().to_task_with_meta(_instance())
+    assert isinstance(task, FeatureDeletionTask)
+    assert meta.diff_reversed is True
+    assert meta.strategy == "lm_modify"
+    # The FIX restores `a + b` (adds it back) and removes the bug.
+    assert "+    return a + b" in task.golden_diff
+    assert "-    return a - b" in task.golden_diff
+    assert UNREVERSED_MARKER not in task.golden_diff
+
+
+def test_to_task_unreversible_patch_gets_marker():
+    inst = _instance(patch="diff --git a/x b/y\nrename from x\nrename to y\n@@ -1 +1 @@\n-a\n+b\n")
+    task, meta = SwesmithAdapter().to_task_with_meta(inst)
+    assert meta.diff_reversed is False
+    assert task.golden_diff.startswith(UNREVERSED_MARKER)
+
+
+def test_image_resolution_prefers_instance_field_then_convention():
+    a = SwesmithAdapter()
+    assert a.image_for(_instance()) == "swesmith.x86_64.getmoto__moto:latest"
+    assert a.image_for(_instance(image_name=None, docker_image="custom:tag")) == "custom:tag"
+    inst = _instance(image_name=None)
+    inst.pop("docker_image", None)
+    assert a.image_for(inst) == "swesmith.x86_64.getmoto__moto:latest"
+
+
+def test_f2p_p2p_tuple_handling_matches_swebench_semantics():
+    task = SwesmithAdapter().to_task(_instance(
+        FAIL_TO_PASS=["t/a.py::t1", "t/a.py::t2"],  # real list, not JSON string
+        PASS_TO_PASS=json.dumps([]),
+    ))
+    assert task.fail_to_pass == ("t/a.py::t1", "t/a.py::t2")
+    assert task.pass_to_pass == ()
+
+
+def test_difficulty_priors_by_strategy():
+    pr = SwesmithAdapter().to_task(_instance(instance_id="r__x.abc.pr_99"))
+    proc = SwesmithAdapter().to_task(_instance(instance_id="r__x.abc.func_pm_remove_loop__1"))
+    assert pr.difficulty_prior < proc.difficulty_prior  # PR Mirror harder prior
+
+
+def test_redistributable_filter_interplay():
+    """repo_gate owns repo-level licensing, but the per-instance filter from
+    SweBenchAdapter still composes when a license field IS present."""
+    task = SwesmithAdapter().to_task(_instance(license_name="GPL-3.0"))
+    assert SweBenchAdapter.is_redistributable(task) is False
+    task2 = SwesmithAdapter().to_task(_instance(license_name="MIT"))
+    assert SweBenchAdapter.is_redistributable(task2) is True
+
+
+def test_load_local_jsonl(tmp_path):
+    p = tmp_path / "fixtures.jsonl"
+    p.write_text("\n".join(json.dumps(_instance(instance_id=f"r__x.abc.pr_{i}")) for i in range(5)))
+    rows = load_swesmith_instances(str(p), limit=3)
+    assert len(rows) == 3
+    assert rows[0]["instance_id"] == "r__x.abc.pr_0"

composer_replication/datagen/tests/test_trajectory.py

@@ -0,0 +1,127 @@
+"""Tests for the canonical trajectory IR (deepread findings V2/D-11/D-8).
+
+The load-bearing test is the SENTINEL leak guard: `to_policy_row` must never
+emit golden_diff/deleted_symbols, even though the task dataclass carries them.
+"""
+from __future__ import annotations
+
+import json
+
+from composer_replication.datagen.schema import FeatureDeletionTask
+from composer_replication.datagen.trajectory import (
+    CanonicalTrajectory,
+    ToolCall,
+    TrajectoryStep,
+    from_trace_states,
+    to_policy_row,
+    to_sft_messages,
+)
+from composer_replication.teacher_replay import TraceState
+
+
+def _task(**over) -> FeatureDeletionTask:
+    base = dict(
+        task_id="t1", repo="org/repo", base_commit="abc",
+        broken_image="img:1", test_command="pytest -q",
+        fail_to_pass=("t/a.py::t1",), pass_to_pass=("t/a.py::t2",),
+        golden_diff="SENTINEL_NEVER_LEAK",
+        deleted_symbols=("secret_fn",),
+    )
+    base.update(over)
+    return FeatureDeletionTask(**base)
+
+
+def _traj() -> CanonicalTrajectory:
+    return CanonicalTrajectory(
+        task_id="t1",
+        steps=[
+            TrajectoryStep(observation="repo is broken", action=ToolCall("bash", {"command": "pytest"}),
+                           result="2 failed", tool_error=False),
+            TrajectoryStep(observation="2 failed", action="here is my final patch",
+                           result="graded", tool_error=False),
+        ],
+        grade=1.0, guard_ok=True, hacked=False,
+        provenance={"source": "test"},
+    )
+
+
+# ---------------------------------------------------------------------
+# ToolCall canonical form — the v1 divergence algebra
+# ---------------------------------------------------------------------
+
+
+def test_canonical_form_is_order_insensitive_on_args():
+    a = ToolCall("edit", {"path": "x.py", "content": "y"})
+    b = ToolCall("edit", {"content": "y", "path": "x.py"})
+    assert a.canonical_form() == b.canonical_form()
+
+
+def test_canonical_form_distinguishes_name_and_args():
+    assert ToolCall("bash", {"command": "ls"}).canonical_form() != \
+        ToolCall("bash", {"command": "ls -la"}).canonical_form()
+    assert ToolCall("read", {"f": "x"}).canonical_form() != \
+        ToolCall("write", {"f": "x"}).canonical_form()
+
+
+# ---------------------------------------------------------------------
+# THE leak guard (finding D-8)
+# ---------------------------------------------------------------------
+
+
+def test_policy_row_never_contains_golden_diff_or_deleted_symbols():
+    row = to_policy_row(_traj(), _task())
+    blob = json.dumps(row)
+    assert "SENTINEL_NEVER_LEAK" not in blob
+    assert "secret_fn" not in blob
+    assert "golden_diff" not in blob
+    assert "deleted_symbols" not in blob
+    # And the row still carries what the policy MAY see.
+    assert row["repo"] == "org/repo"
+    assert row["fail_to_pass"] == ["t/a.py::t1"]
+    assert row["grade"] == 1.0
+
+
+# ---------------------------------------------------------------------
+# IR ↔ SFT messages
+# ---------------------------------------------------------------------
+
+
+def test_to_sft_messages_alternates_roles():
+    msgs = to_sft_messages(_traj())
+    assert msgs[0] == {"role": "user", "content": "repo is broken"}
+    assert msgs[1]["role"] == "assistant"
+    assert "[TOOL_USE] name=bash" in msgs[1]["content"]
+    assert msgs[2] == {"role": "user", "content": "2 failed"}
+
+
+# ---------------------------------------------------------------------
+# Claude Code → IR adapter
+# ---------------------------------------------------------------------
+
+
+def test_from_trace_states_parses_single_tool_use_and_error_flag():
+    states: list[TraceState] = [
+        {
+            "state_id": "sess1::0000",
+            "messages": [
+                {"role": "system", "content": "sys"},
+                {"role": "user", "content": "[TOOL_RESULT (ERROR)] (id=x)\nboom",
+                 "tool_error": True},
+            ],
+            "student_action": '[TOOL_USE] name=Bash input={"command":"ls"}',
+        },
+        {
+            "state_id": "sess1::0001",
+            "messages": [{"role": "user", "content": "plain prompt"}],
+            "student_action": "I think the fix is...\n\n[TOOL_USE] name=Edit input={\"p\":1}\n\n[TOOL_USE] name=Bash input={\"c\":2}",
+        },
+    ]
+    traj = from_trace_states(states)
+    assert traj.task_id == "sess1"
+    assert traj.grade is None  # ungraded — Claude Code traces have no oracle
+    s0, s1 = traj.steps
+    assert isinstance(s0.action, ToolCall) and s0.action.name == "Bash"
+    assert s0.tool_error is True
+    # Multi-tool turn stays as the raw string (honest, not guessed).
+    assert isinstance(s1.action, str)
+    assert s1.tool_error is False

composer_replication/datagen/trajectory.py

@@ -0,0 +1,203 @@
+"""trajectory.py — the canonical trajectory IR (deepread findings V2/D-11/D-8).
+
+THE GAP THIS CLOSES: the repo had 3 (heading to 5) incompatible trajectory
+shapes — Claude Code TraceState text serialization, Bedrock `.jsonl.out` rows,
+the planned tree/rollout/OpenHands shapes — with no shared schema, and the
+divergence gate rested on a whitespace-collapse string normalizer
+(`teacher_replay._normalize_action`, self-admitted skeleton). This module is the
+single intermediate representation every producer adapts INTO and every corpus
+writer reads FROM.
+
+SECURITY INVARIANT (finding D-8): `FeatureDeletionTask.golden_diff` uses
+`repr=False`, but `dataclasses.asdict()` and naive JSON serialization still
+include it. `to_policy_row()` is the ONE serializer allowed to produce
+policy-visible rows, and its output is unit-tested to never contain
+`golden_diff` or `deleted_symbols`.
+"""
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass, field
+from typing import Any
+
+from composer_replication.datagen.schema import FeatureDeletionTask
+from composer_replication.teacher_replay import TraceState
+
+#: Bump when the IR shape changes; carried on every trajectory + corpus row.
+SCHEMA_VERSION = "1"
+
+
+@dataclass(frozen=True)
+class ToolCall:
+    """One structured tool invocation — the unit of the action algebra.
+
+    `canonical_form()` is the v1 divergence-gate algebra (finding D-3): two
+    actions are "the same" iff their canonical forms match (tool name + sorted,
+    JSON-normalized args). This replaces the whitespace-collapse stub that made
+    the divergence gate fire on noise. v2 will need arg-level normalization
+    (path equivalence, whitespace-insensitive code args); keep that evolution
+    HERE so every consumer inherits it.
+    """
+
+    name: str
+    args: dict = field(default_factory=dict)
+
+    def canonical_form(self) -> str:
+        try:
+            args_json = json.dumps(self.args, sort_keys=True, separators=(",", ":"))
+        except (TypeError, ValueError):
+            args_json = str(self.args)
+        return f"{self.name}:{args_json}"
+
+
+@dataclass
+class TrajectoryStep:
+    """One agent turn: what it saw, what it did, what came back."""
+
+    observation: str
+    action: ToolCall | str          # str = plain text / final message
+    result: str | None = None      # tool output observed AFTER the action
+    tool_error: bool = False
+
+
+@dataclass
+class CanonicalTrajectory:
+    """The IR: an episode (or trace) as a typed step list + outcome + provenance."""
+
+    task_id: str
+    steps: list[TrajectoryStep] = field(default_factory=list)
+    grade: float | None = None      # _grade() pass-fraction; None = ungraded trace
+    guard_ok: bool = True
+    hacked: bool = False
+    provenance: dict = field(default_factory=dict)  # source, policy id, cost_usd, run_id
+    schema_version: str = SCHEMA_VERSION
+
+
+# ---------------------------------------------------------------------
+# Producers → IR
+# ---------------------------------------------------------------------
+
+# ClaudeCodeIngester serializes tool calls as "[TOOL_USE] name=<n> input=<json>".
+_TOOL_USE_RE = re.compile(r"^\[TOOL_USE\] name=(?P<name>\S+) input=(?P<input>\{.*\})$")
+
+
+def _parse_action(student_action: str) -> ToolCall | str:
+    """Parse a TraceState student_action back into a ToolCall where possible.
+
+    Claude Code assistant turns serialize as newline-joined blocks; if exactly
+    one [TOOL_USE] block is present we recover the structured call (the common
+    case ADR-002 chose one-node-per-turn for). Multi-tool turns and pure-text /
+    thinking turns stay as the raw string — honest about what we can't
+    structure rather than guessing.
+    """
+    blocks = [b for b in student_action.split("\n\n") if b.strip()]
+    tool_blocks = [b for b in blocks if b.startswith("[TOOL_USE]")]
+    if len(tool_blocks) == 1:
+        m = _TOOL_USE_RE.match(tool_blocks[0].strip())
+        if m:
+            try:
+                return ToolCall(name=m.group("name"), args=json.loads(m.group("input")))
+            except (json.JSONDecodeError, ValueError):
+                pass
+    return student_action
+
+
+def from_trace_states(
+    states: list[TraceState],
+    *,
+    task_id: str = "",
+    provenance: dict | None = None,
+) -> CanonicalTrajectory:
+    """Adapt a Claude Code trace (TraceState list) into the IR.
+
+    HONEST CAPABILITY NOTE (finding D-1): these traces carry no executable
+    environment — no broken_image, no FAIL_TO_PASS — so the resulting
+    trajectory is UNGRADED (`grade=None`) and is admissible only for flat
+    Channel-3 / SFT-style uses, never as a tree seed. Env-grounded rollouts
+    (rollout_harness.collect_trajectory) are the graded producers.
+    """
+    steps: list[TrajectoryStep] = []
+    for s in states:
+        # The observation for step t is the last user message before the turn.
+        obs = ""
+        tool_error = False
+        for msg in reversed(s["messages"]):
+            if msg.get("role") == "user":
+                obs = str(msg.get("content", ""))
+                tool_error = bool(msg.get("tool_error", False))
+                break
+        steps.append(TrajectoryStep(
+            observation=obs,
+            action=_parse_action(s["student_action"]),
+            result=None,
+            tool_error=tool_error,
+        ))
+    prov = {"source": "claude_code", **(provenance or {})}
+    return CanonicalTrajectory(task_id=task_id or (states[0]["state_id"].split("::")[0] if states else ""),
+                               steps=steps, grade=None, provenance=prov)
+
+
+# ---------------------------------------------------------------------
+# IR → consumers
+# ---------------------------------------------------------------------
+
+
+def _action_text(action: ToolCall | str) -> str:
+    if isinstance(action, ToolCall):
+        return f"[TOOL_USE] name={action.name} input=" + json.dumps(
+            action.args, separators=(",", ":")
+        )
+    return action
+
+
+def to_sft_messages(traj: CanonicalTrajectory) -> list[dict]:
+    """IR → OpenAI-style messages for SFT (obs→user, action→assistant)."""
+    messages: list[dict] = []
+    for step in traj.steps:
+        if step.observation:
+            messages.append({"role": "user", "content": step.observation})
+        messages.append({"role": "assistant", "content": _action_text(step.action)})
+        if step.result:
+            messages.append({"role": "user", "content": step.result})
+    return messages
+
+
+#: Task fields the POLICY may see. Everything else (golden_diff,
+#: deleted_symbols) is construction-side and must never reach a corpus row.
+_POLICY_VISIBLE_TASK_FIELDS = (
+    "task_id", "repo", "base_commit", "test_command",
+    "fail_to_pass", "pass_to_pass", "granularity", "difficulty_prior",
+)
+
+
+def to_policy_row(traj: CanonicalTrajectory, task: FeatureDeletionTask) -> dict:
+    """THE policy-visible corpus serializer (finding D-8 — the leak guard).
+
+    Builds the row field-by-field from an allowlist; never `asdict(task)`,
+    which would include `golden_diff` despite its `repr=False`. Unit-tested
+    with a sentinel to prove the absence.
+    """
+    row: dict[str, Any] = {
+        "schema_version": traj.schema_version,
+        "messages": to_sft_messages(traj),
+        "grade": traj.grade,
+        "guard_ok": traj.guard_ok,
+        "hacked": traj.hacked,
+        "provenance": dict(traj.provenance),
+    }
+    for f in _POLICY_VISIBLE_TASK_FIELDS:
+        v = getattr(task, f)
+        row[f] = list(v) if isinstance(v, tuple) else v
+    return row
+
+
+__all__ = [
+    "SCHEMA_VERSION",
+    "ToolCall",
+    "TrajectoryStep",
+    "CanonicalTrajectory",
+    "from_trace_states",
+    "to_sft_messages",
+    "to_policy_row",
+]

composer_replication/diloco/__init__.py

@@ -4,9 +4,12 @@ Wraps `torchft.local_sgd.DiLoCo` with the framework's conventions:
 - Sign convention is documented LOUDLY here once and tested via Spike 008.
 - The wrapper exposes the same constructor shape as torchft's DiLoCo so a
   future swap-in of the upstream class is a one-line change.
-- Vanilla DiLoCo (Douillard et al. 2023) = `fragment_sync_delay=0`, single
-  fragment. Streaming DiLoCo (Liu et al. 2025) = non-zero delay, multiple
-  fragments. Spike 008 uses vanilla; Streaming is configured by the same API.
+- Vanilla DiLoCo (Douillard et al. 2023, arXiv:2311.08105) =
+  `fragment_sync_delay=0`, single fragment. Streaming DiLoCo (Douillard et
+  al., arXiv:2501.18512 "Streaming DiLoCo with overlapping communication";
+  the separate Eager-Updates work is Kale et al., arXiv:2502.12996 — citation
+  corrected per deepread finding V7) = non-zero delay, multiple fragments.
+  Spike 008 uses vanilla; Streaming is configured by the same API.
 
 Reference: `docs/adrs/ADR-003-diloco-impl.md`.
 

composer_replication/opsd.py

@@ -10,17 +10,23 @@ Mathematical reference:
 - OPSD paper: Zhao et al., "Self-Distilled Reasoner: On-Policy Self-Distillation
   for LLMs", arXiv:2601.18734.
 - SDPO paper: Hübotter et al., "Reinforcement Learning via Self-Distillation",
-  arXiv:2601.20802 (formalizes the same loss as Composer 2.5's "Targeted RL with
-  Textual Feedback").
+  arXiv:2601.20802. PROVENANCE (corrected per deepread finding V1): Cursor's
+  blog cites SDPO/OPSD only as *background* ("For more background on this
+  approach see…"), NOT as its mechanism. Published SDPO distills over the FULL
+  rollout with feedback in the prefix and an EMA-regularized teacher; this
+  repo's channel is a turn-localized hint-splice with a live (stop-grad,
+  non-EMA) teacher — a third, blog-inspired design, neither verbatim SDPO nor
+  confirmed-Composer. The kernel below matches OPSD's generalized JSD math.
 
 The loss computes JSD/KL divergence between a teacher distribution (model
 conditioned on privileged information / a hint) and a student distribution
 (model on the original context). Both come from the SAME model — the teacher
 is just "the model with hint inserted into context."
 
-Composer 2.5 uses this with the privileged information being a "hint" inserted
-at the error-turn site. We use the same loss; the data collator constructs
-ctx_teacher = ctx_student + hint_at_error_turn for us.
+Composer 2.5's blog describes inserting a "hint" at the error-turn site and
+distilling the student toward the hint-conditioned distribution "for that turn
+only". The data collator constructs ctx_teacher = ctx_student +
+hint_at_error_turn for us.
 """
 
 from __future__ import annotations

composer_replication/pipeline/__init__.py

@@ -0,0 +1,38 @@
+"""composer_replication.pipeline — the Stage-0 dataset-pipeline contract + driver.
+
+THE single reconciled dataset contract (supersedes the two divergent layouts in
+research/design-F1 and design-F2 — deepread finding V8/D-7), the pragmatic
+near-duplicate detector, and the local stage-driver that turns
+(tasks, env, policy) into a carded, deduped, holdout-split corpus.
+"""
+from composer_replication.pipeline.build_corpus import build_corpus
+from composer_replication.pipeline.dedup import (
+    dedup,
+    find_near_duplicates,
+    jaccard_estimate,
+    minhash_signature,
+)
+from composer_replication.pipeline.s3_contract import (
+    RunLayout,
+    RunManifest,
+    write_dataset_card,
+    write_dpo_rows,
+    write_sft_rows,
+    write_tasks,
+    write_tasks_full,
+)
+
+__all__ = [
+    "RunLayout",
+    "RunManifest",
+    "build_corpus",
+    "dedup",
+    "find_near_duplicates",
+    "jaccard_estimate",
+    "minhash_signature",
+    "write_dataset_card",
+    "write_dpo_rows",
+    "write_sft_rows",
+    "write_tasks",
+    "write_tasks_full",
+]

composer_replication/pipeline/build_corpus.py

@@ -0,0 +1,137 @@
+"""build_corpus.py — the local Stage-0 stage-driver (architecture step 6-7).
+
+One function wires the whole local pipeline: holdout-split the task pool
+(holdout tasks are NEVER rolled out — they are the eval anchor), roll out a
+policy over each train task, admit + type + route trajectories
+(sft / dpo-candidate / quarantine), dedup the SFT rows (within-run AND against
+a prior generation's signatures), and write everything through the single
+s3_contract layout with a manifest + dataset card.
+
+Deliberately LOCAL-first (finding D-9): the five-service AWS orchestration is
+Stage 4; this driver must produce one real corpus end-to-end on a laptop with a
+FakeSandbox before anything is distributed. Write-once per layout
+(finding D-21): refuses to run if the manifest already exists.
+"""
+from __future__ import annotations
+
+from typing import Callable, Sequence
+
+from composer_replication.datagen.env import FeatureDeletionEnv
+from composer_replication.datagen.rollout_harness import (
+    RolloutPolicy,
+    admit,
+    collect_trajectory,
+)
+from composer_replication.datagen.schema import FeatureDeletionTask
+from composer_replication.datagen.trajectory import to_policy_row
+from composer_replication.pipeline import s3_contract
+from composer_replication.pipeline.dedup import dedup
+from composer_replication.pipeline.s3_contract import RunLayout, RunManifest
+from composer_replication.safety.holdout import HeldoutSplit
+
+
+def build_corpus(
+    source_tasks: Sequence[FeatureDeletionTask],
+    env_factory: Callable[[], FeatureDeletionEnv],
+    policy_factory: Callable[[], RolloutPolicy],
+    layout: RunLayout,
+    manifest: RunManifest,
+    *,
+    holdout_frac: float = 0.2,
+    holdout_seed: int = 0,
+    max_tasks: int | None = None,
+    cost_per_rollout_usd: float = 0.0,
+    prior_signatures: Sequence[Sequence[int]] | None = None,
+    dedup_threshold: float = 0.85,
+) -> RunManifest:
+    """Run the Stage-0 pipeline over `source_tasks`; returns the final manifest.
+
+    Args:
+        source_tasks: gate_repo-admitted FeatureDeletionTasks (the caller runs
+            `datagen.repo_gate.gate_repo` BEFORE this — the driver assumes the
+            license/decontamination gates already passed).
+        env_factory: fresh `FeatureDeletionEnv` per rollout (a sandbox is
+            stateful; sharing one across episodes leaks trajectory state).
+        policy_factory: fresh policy per rollout (ScriptedPolicy is stateful).
+        manifest: a `RunManifest` with run_id/created_at/budget set by the
+            caller (created_at is caller-passed for reproducibility).
+        cost_per_rollout_usd: accounting hook — API policies should report
+            real cost; the driver enforces `manifest.budget_usd` with it.
+        prior_signatures: previous generation's MinHash signatures
+            (cross-generation dedup, finding D-12).
+
+    Raises:
+        FileExistsError: if the layout already has a manifest (write-once).
+    """
+    if s3_contract.manifest_exists(layout):
+        raise FileExistsError(
+            f"Run layout already has a manifest at {layout.manifest_path} — "
+            "runs are write-once per (root, run_id); mint a new run_id "
+            "(finding D-21 idempotency)."
+        )
+
+    # 1. Holdout split FIRST — held-out tasks are never rolled out, so no
+    #    training signal can derive from them (the HeldoutSplit discipline).
+    split = HeldoutSplit.split(source_tasks, holdout_frac=holdout_frac,
+                               seed=holdout_seed, check_content=True)
+    by_id = {t.task_id: t for t in source_tasks}
+    holdout_tasks = [by_id[i] for i in sorted(split.holdout_ids)]
+    train_tasks = [by_id[i] for i in sorted(split.train_ids)]
+    if max_tasks is not None:
+        train_tasks = train_tasks[:max_tasks]
+
+    # 2. Rollouts + admission routing.
+    sft_rows: list[dict] = []
+    dpo_rows: list[dict] = []
+    quarantine_rows: list[dict] = []
+    traj_rows: list[dict] = []
+    partial = False
+    for task in train_tasks:
+        if manifest.over_budget:
+            partial = True
+            break
+        traj = collect_trajectory(env_factory(), task, policy_factory(),
+                                  provenance={"run_id": manifest.run_id})
+        manifest.spend(cost_per_rollout_usd)
+        verdict = admit(traj)
+        row = to_policy_row(traj, task)
+        traj_rows.append({**row, "admission": list(verdict.reasons)})
+        if verdict.sft_admitted:
+            sft_rows.append(row)
+        elif verdict.dpo_candidate:
+            dpo_rows.append(row)
+        else:
+            quarantine_rows.append({**row, "reasons": list(verdict.reasons)})
+
+    # 3. Dedup the SFT corpus (within-run + cross-generation).
+    def _key(r: dict) -> str:
+        return " ".join(m.get("content", "") for m in r.get("messages", []))
+
+    sft_rows, dedup_stats = dedup(sft_rows, _key, dedup_threshold,
+                                  prior_signatures=prior_signatures)
+
+    # 4. Write everything through the contract.
+    s3_contract.write_tasks(layout, train_tasks)
+    s3_contract.write_tasks_full(layout, train_tasks)
+    s3_contract.write_holdout(layout, holdout_tasks)
+    s3_contract.write_trajectories(layout, traj_rows)
+    s3_contract.write_sft_rows(layout, sft_rows)
+    s3_contract.write_dpo_rows(layout, dpo_rows)
+    s3_contract.write_quarantine(layout, quarantine_rows)
+
+    manifest.counts = {
+        "tasks_train": len(train_tasks),
+        "tasks_holdout": len(holdout_tasks),
+        "rollouts": len(traj_rows),
+        "sft_rows": len(sft_rows),
+        "dpo_rows": len(dpo_rows),
+        "quarantined": len(quarantine_rows),
+        **{f"dedup_{k}": v for k, v in dedup_stats.items()},
+    }
+    manifest.status = "partial" if partial else "building"
+    manifest.write(layout)
+    s3_contract.write_dataset_card(layout, manifest, dedup_stats=dedup_stats)
+    return manifest
+
+
+__all__ = ["build_corpus"]

composer_replication/pipeline/dedup.py

@@ -0,0 +1,138 @@
+"""dedup.py — MinHash near-duplicate detection (finding D-12).
+
+Cross-generation dedup is a flywheel-collapse mitigation: a self-training loop
+that re-ingests its own outputs accumulates near-identical rows, and per-batch
+`document_deduplicator` (the only dedup the old designs had) never sees across
+runs. This module computes MinHash signatures over word 5-shingles so a run can
+(a) dedup within itself and (b) accept the PRIOR run's signature file and dedup
+against it (lineage threaded by `RunManifest.parent_run_id`).
+
+Pragmatic v1: builtin-hash permutation MinHash with N=64 seeds, no banding/LSH
+(O(n^2) pair scan — fine for Stage-0 corpus sizes, thousands of rows).
+`datasketch` (MinHashLSH) is the documented upgrade path when row counts make
+the pair scan bite.
+
+NOTE on hash stability: Python's builtin `hash()` over str is salted per
+process (PYTHONHASHSEED), which would make signatures non-portable across
+runs — exactly what cross-generation dedup needs. We therefore use md5-based
+hashing (stable everywhere) despite the small speed cost.
+"""
+from __future__ import annotations
+
+import hashlib
+import json
+import re
+from typing import IO, Callable, Iterable, Sequence
+
+N_PERMUTATIONS = 64
+_SHINGLE_W = 5
+_WORD_RE = re.compile(r"\w+")
+_MAX64 = (1 << 64) - 1
+
+
+def _shingles(text: str, w: int = _SHINGLE_W) -> set[str]:
+    words = _WORD_RE.findall(text.lower())
+    if len(words) <= w:
+        return {" ".join(words)} if words else set()
+    return {" ".join(words[i:i + w]) for i in range(len(words) - w + 1)}
+
+
+def _stable_hash(s: str, seed: int) -> int:
+    h = hashlib.md5(f"{seed}:{s}".encode()).digest()
+    return int.from_bytes(h[:8], "big")
+
+
+def minhash_signature(text: str, n_perm: int = N_PERMUTATIONS) -> tuple[int, ...]:
+    """MinHash signature: per-seed minimum over the shingle set."""
+    sh = _shingles(text)
+    if not sh:
+        return tuple([_MAX64] * n_perm)
+    return tuple(min(_stable_hash(s, seed) for s in sh) for seed in range(n_perm))
+
+
+def jaccard_estimate(sig_a: Sequence[int], sig_b: Sequence[int]) -> float:
+    """Estimated Jaccard similarity = fraction of agreeing signature slots."""
+    if len(sig_a) != len(sig_b) or not sig_a:
+        raise ValueError("signatures must be equal-length and non-empty")
+    return sum(1 for a, b in zip(sig_a, sig_b) if a == b) / len(sig_a)
+
+
+def find_near_duplicates(
+    rows: Sequence[dict],
+    key_fn: Callable[[dict], str],
+    threshold: float = 0.85,
+    *,
+    prior_signatures: Sequence[Sequence[int]] | None = None,
+) -> list[tuple[int, int]]:
+    """All (i, j) index pairs whose estimated Jaccard >= threshold.
+
+    `prior_signatures` (from a previous run) participate as virtual rows with
+    negative indices -(k+1), so a pair (i, -1) means "row i duplicates prior
+    signature 0" — the cross-generation case.
+    """
+    sigs = [minhash_signature(key_fn(r)) for r in rows]
+    pairs: list[tuple[int, int]] = []
+    for i in range(len(sigs)):
+        for j in range(i + 1, len(sigs)):
+            if jaccard_estimate(sigs[i], sigs[j]) >= threshold:
+                pairs.append((i, j))
+        for k, prior in enumerate(prior_signatures or []):
+            if jaccard_estimate(sigs[i], prior) >= threshold:
+                pairs.append((i, -(k + 1)))
+    return pairs
+
+
+def dedup(
+    rows: Sequence[dict],
+    key_fn: Callable[[dict], str],
+    threshold: float = 0.85,
+    *,
+    prior_signatures: Sequence[Sequence[int]] | None = None,
+) -> tuple[list[dict], dict]:
+    """Keep-first dedup. Returns (kept_rows, stats).
+
+    A row duplicating a PRIOR-run signature is dropped outright (the prior run
+    already owns it); within-run duplicates keep the earliest occurrence.
+    """
+    pairs = find_near_duplicates(rows, key_fn, threshold,
+                                 prior_signatures=prior_signatures)
+    drop: set[int] = set()
+    for i, j in pairs:
+        if j < 0:
+            drop.add(i)          # duplicates a prior-run row
+        else:
+            drop.add(j)          # keep-first within this run
+    kept = [r for i, r in enumerate(rows) if i not in drop]
+    return kept, {
+        "rows_in": len(rows),
+        "rows_kept": len(kept),
+        "dropped_within_run": len({j for _, j in pairs if j >= 0} & drop),
+        "dropped_cross_generation": len({i for i, j in pairs if j < 0} & drop),
+        "threshold": threshold,
+    }
+
+
+def signatures_to_jsonl(rows: Sequence[dict], key_fn: Callable[[dict], str],
+                        fh: IO[str]) -> int:
+    """Persist this run's signatures so the NEXT generation can dedup against
+    them (pass the loaded list as `prior_signatures`)."""
+    n = 0
+    for r in rows:
+        fh.write(json.dumps(list(minhash_signature(key_fn(r)))) + "\n")
+        n += 1
+    return n
+
+
+def load_signatures(fh: IO[str]) -> list[tuple[int, ...]]:
+    return [tuple(json.loads(line)) for line in fh if line.strip()]
+
+
+__all__ = [
+    "N_PERMUTATIONS",
+    "dedup",
+    "find_near_duplicates",
+    "jaccard_estimate",
+    "load_signatures",
+    "minhash_signature",
+    "signatures_to_jsonl",
+]

composer_replication/pipeline/s3_contract.py

@@ -0,0 +1,287 @@
+"""s3_contract.py — THE single dataset layout + manifest (finding V8/D-7/D-8).
+
+Supersedes BOTH prior contracts: design-F1's `runs/<id>/{sft_corpus,dpo_pairs,
+rl_task_pool,divergence_pairs,wm_tuples,holdout,diloco_rendezvous}` and
+design-F2's `{traces,tasks,replay,task_grades,corpus}/v1/run_id=<id>` — the two
+were never reconciled and coexisted in the grounding doc. One layout, one
+manifest, two explicit serializers with a unit-tested leak guard.
+
+Deliberate exclusions from the run layout:
+  * `diloco_rendezvous/` — training-comms state, not dataset; lives in its own
+    prefix/bucket (finding D-19).
+  * `wm_tuples/` — emitted only when the P4 world-model ablation is scheduled
+    (finding D-14); not part of Stage 0.
+
+Layout (root = any local path or fsspec URI):
+    <root>/runs/<run_id>/
+        tasks/manifest.jsonl       policy-safe task rows (golden_diff -> sha256)
+        tasks_full/manifest.jsonl  construction-side full rows (RESTRICTED prefix)
+        traj/*.jsonl               CanonicalTrajectory records (audit trail)
+        corpus_sft/rows.jsonl      admitted SFT rows (to_policy_row output)
+        corpus_dpo/rows.jsonl      DPO-candidate rows
+        holdout/tasks.jsonl        held-out task ids+rows (never rolled out)
+        quarantine/*.jsonl         rejected trajectories w/ reasons (audit)
+        manifest.json              RunManifest
+        DATASET_CARD.md            human-readable card
+"""
+from __future__ import annotations
+
+import dataclasses
+import hashlib
+import json
+from dataclasses import dataclass, field
+from typing import IO, Iterable
+
+from composer_replication.datagen.schema import FeatureDeletionTask
+
+SCHEMA_VERSION = "1"
+
+
+def _is_local(root: str) -> bool:
+    return "://" not in root or root.startswith("file://")
+
+
+def _open(path: str, mode: str = "w") -> IO[str]:
+    """Open a path for text IO; plain `open` locally, fsspec for s3:// etc.
+
+    fsspec is lazy so the module (and all local-corpus runs) need no extra dep.
+    """
+    if _is_local(path):
+        import os
+        local = path.removeprefix("file://")
+        os.makedirs(os.path.dirname(local), exist_ok=True)
+        return open(local, mode, encoding="utf-8")
+    try:
+        import fsspec  # noqa: PLC0415 — lazy heavy dep
+    except ImportError as e:
+        raise RuntimeError(
+            "Non-local corpus roots require fsspec; install with "
+            "`pip install -e .[serverless]`. Got: " + repr(e)
+        ) from e
+    return fsspec.open(path, mode, encoding="utf-8").open()
+
+
+def _exists(path: str) -> bool:
+    if _is_local(path):
+        import os
+        return os.path.exists(path.removeprefix("file://"))
+    import fsspec  # noqa: PLC0415
+    fs, _, paths = fsspec.get_fs_token_paths(path)
+    return bool(fs.exists(paths[0]))
+
+
+@dataclass(frozen=True)
+class RunLayout:
+    """Pure-path logic for one run's prefixes — testable without any IO."""
+
+    root: str
+    run_id: str
+
+    def _p(self, *parts: str) -> str:
+        base = self.root.rstrip("/")
+        return f"{base}/runs/{self.run_id}/" + "/".join(parts)
+
+    @property
+    def tasks_path(self) -> str:
+        return self._p("tasks", "manifest.jsonl")
+
+    @property
+    def tasks_full_path(self) -> str:
+        # RESTRICTED prefix: carries golden_diff/deleted_symbols. On S3 this
+        # prefix gets a deny-by-default policy; locally it is still separated
+        # so a naive `corpus_*` glob can never sweep it up.
+        return self._p("tasks_full", "manifest.jsonl")
+
+    @property
+    def traj_path(self) -> str:
+        return self._p("traj", "trajectories.jsonl")
+
+    @property
+    def sft_path(self) -> str:
+        return self._p("corpus_sft", "rows.jsonl")
+
+    @property
+    def dpo_path(self) -> str:
+        return self._p("corpus_dpo", "rows.jsonl")
+
+    @property
+    def holdout_path(self) -> str:
+        return self._p("holdout", "tasks.jsonl")
+
+    @property
+    def quarantine_path(self) -> str:
+        return self._p("quarantine", "rejected.jsonl")
+
+    @property
+    def manifest_path(self) -> str:
+        return self._p("manifest.json")
+
+    @property
+    def card_path(self) -> str:
+        return self._p("DATASET_CARD.md")
+
+
+@dataclass
+class RunManifest:
+    """Run-level metadata: counts, cost, lineage, budget, acceptance status.
+
+    `created_at` is CALLER-passed (never datetime.now() in here) so manifests
+    are reproducible in tests. `parent_run_id` threads flywheel lineage so
+    cross-generation dedup (finding D-12) can find prior signatures.
+    """
+
+    run_id: str
+    created_at: str
+    source: str = ""
+    counts: dict = field(default_factory=dict)
+    cost_usd: float = 0.0
+    parent_run_id: str | None = None
+    schema_version: str = SCHEMA_VERSION
+    status: str = "building"          # building | accepted | rejected | partial
+    budget_usd: float | None = None
+
+    def spend(self, usd: float) -> None:
+        self.cost_usd += usd
+
+    @property
+    def over_budget(self) -> bool:
+        return self.budget_usd is not None and self.cost_usd >= self.budget_usd
+
+    def write(self, layout: RunLayout) -> None:
+        with _open(layout.manifest_path) as f:
+            json.dump(dataclasses.asdict(self), f, indent=2)
+
+    @classmethod
+    def read(cls, layout: RunLayout) -> RunManifest:
+        with _open(layout.manifest_path, "r") as f:
+            return cls(**json.load(f))
+
+
+# ---------------------------------------------------------------------
+# Writers — the leak guard lives here (finding D-8)
+# ---------------------------------------------------------------------
+
+
+def _task_row_policy_safe(task: FeatureDeletionTask) -> dict:
+    """Task row with the construction-side secrets REPLACED, not just hidden.
+
+    `asdict()` includes `golden_diff` despite `repr=False` — that is exactly
+    the leak D-8 flagged. We keep provenance via a sha256 (verifiable, not
+    recoverable) and drop `deleted_symbols` entirely (they name the answer).
+    """
+    row = dataclasses.asdict(task)
+    gold = row.pop("golden_diff", "")
+    row.pop("deleted_symbols", None)
+    row["golden_diff_sha256"] = hashlib.sha256(gold.encode()).hexdigest() if gold else ""
+    return row
+
+
+def write_tasks(layout: RunLayout, tasks: Iterable[FeatureDeletionTask]) -> int:
+    """Write the POLICY-SAFE task manifest (the default everything reads)."""
+    n = 0
+    with _open(layout.tasks_path) as f:
+        for t in tasks:
+            f.write(json.dumps(_task_row_policy_safe(t)) + "\n")
+            n += 1
+    return n
+
+
+def write_tasks_full(layout: RunLayout, tasks: Iterable[FeatureDeletionTask]) -> int:
+    """Write FULL task rows (incl. golden_diff) to the RESTRICTED prefix.
+
+    Only the validator/monitor side reads this; never corpus consumers.
+    """
+    n = 0
+    with _open(layout.tasks_full_path) as f:
+        for t in tasks:
+            f.write(json.dumps(dataclasses.asdict(t)) + "\n")
+            n += 1
+    return n
+
+
+def _write_jsonl(path: str, rows: Iterable[dict]) -> int:
+    n = 0
+    with _open(path) as f:
+        for r in rows:
+            f.write(json.dumps(r) + "\n")
+            n += 1
+    return n
+
+
+def write_sft_rows(layout: RunLayout, rows: Iterable[dict]) -> int:
+    return _write_jsonl(layout.sft_path, rows)
+
+
+def write_dpo_rows(layout: RunLayout, rows: Iterable[dict]) -> int:
+    return _write_jsonl(layout.dpo_path, rows)
+
+
+def write_quarantine(layout: RunLayout, rows: Iterable[dict]) -> int:
+    return _write_jsonl(layout.quarantine_path, rows)
+
+
+def write_holdout(layout: RunLayout, tasks: Iterable[FeatureDeletionTask]) -> int:
+    return _write_jsonl(layout.holdout_path, (_task_row_policy_safe(t) for t in tasks))
+
+
+def write_trajectories(layout: RunLayout, rows: Iterable[dict]) -> int:
+    return _write_jsonl(layout.traj_path, rows)
+
+
+def write_dataset_card(layout: RunLayout, manifest: RunManifest,
+                       *, license_tiers: dict[str, int] | None = None,
+                       dedup_stats: dict | None = None,
+                       decontamination_note: str = "") -> None:
+    """A small human-readable dataset card (finding D-18)."""
+    lines = [
+        f"# Dataset card — run `{manifest.run_id}`",
+        "",
+        f"- **created:** {manifest.created_at}",
+        f"- **source:** {manifest.source}",
+        f"- **status:** {manifest.status}",
+        f"- **schema_version:** {manifest.schema_version}",
+        f"- **cost (USD):** {manifest.cost_usd:.2f}"
+        + (f" / budget {manifest.budget_usd:.2f}" if manifest.budget_usd else ""),
+        f"- **lineage:** parent_run_id={manifest.parent_run_id or 'none'}",
+        "",
+        "## Counts",
+        "",
+    ]
+    for k, v in sorted(manifest.counts.items()):
+        lines.append(f"- {k}: {v}")
+    if license_tiers:
+        lines += ["", "## License tiers seen", ""]
+        lines += [f"- {k}: {v}" for k, v in sorted(license_tiers.items())]
+    lines += ["", "## Decontamination", "",
+              decontamination_note or
+              "All source repos checked against the SWE-bench-family eval list "
+              "(datagen.repo_gate.DECONTAMINATION_LIST) at ingest."]
+    if dedup_stats:
+        lines += ["", "## Dedup", ""]
+        lines += [f"- {k}: {v}" for k, v in sorted(dedup_stats.items())]
+    lines += ["", "Policy-safe rows only: `golden_diff` is sha256-hashed and "
+              "`deleted_symbols` dropped in `tasks/`, `corpus_*/`, `holdout/` "
+              "(full rows live in the restricted `tasks_full/`).", ""]
+    with _open(layout.card_path) as f:
+        f.write("\n".join(lines))
+
+
+def manifest_exists(layout: RunLayout) -> bool:
+    """Write-once guard for the driver (finding D-21 idempotency)."""
+    return _exists(layout.manifest_path)
+
+
+__all__ = [
+    "SCHEMA_VERSION",
+    "RunLayout",
+    "RunManifest",
+    "manifest_exists",
+    "write_dataset_card",
+    "write_dpo_rows",
+    "write_holdout",
+    "write_quarantine",
+    "write_sft_rows",
+    "write_tasks",
+    "write_tasks_full",
+    "write_trajectories",
+]

composer_replication/pipeline/tests/test_pipeline.py

@@ -0,0 +1,223 @@
+"""Tests for the Stage-0 pipeline: contract, dedup, driver.
+
+Load-bearing coverage: the sentinel leak guard on write_tasks (finding D-8),
+holdout exclusion + budget stop + idempotency in build_corpus (D-21), and the
+cross-generation dedup path (D-12).
+"""
+from __future__ import annotations
+
+import io
+import json
+from pathlib import Path
+
+import pytest
+
+from composer_replication.datagen.env import FeatureDeletionEnv
+from composer_replication.datagen.rollout_harness import ScriptedPolicy
+from composer_replication.datagen.sandbox import FakeSandbox
+from composer_replication.datagen.schema import FeatureDeletionTask
+from composer_replication.datagen.trajectory import ToolCall
+from composer_replication.pipeline.build_corpus import build_corpus
+from composer_replication.pipeline.dedup import (
+    dedup,
+    find_near_duplicates,
+    jaccard_estimate,
+    load_signatures,
+    minhash_signature,
+    signatures_to_jsonl,
+)
+from composer_replication.pipeline.s3_contract import (
+    RunLayout,
+    RunManifest,
+    write_dataset_card,
+    write_tasks,
+    write_tasks_full,
+)
+
+
+def _task(i: int, **over) -> FeatureDeletionTask:
+    base = dict(
+        task_id=f"task-{i:03d}", repo="org/repo", base_commit="abc",
+        broken_image="img:1", test_command="pytest -q",
+        fail_to_pass=(f"t/a.py::t{i}",), pass_to_pass=("t/a.py::keep",),
+        golden_diff="SENTINEL_NEVER_LEAK", deleted_symbols=("secret_fn",),
+    )
+    base.update(over)
+    return FeatureDeletionTask(**base)
+
+
+# ---------------------------------------------------------------------
+# RunLayout / RunManifest
+# ---------------------------------------------------------------------
+
+
+def test_layout_paths_are_pure_and_namespaced():
+    lay = RunLayout(root="/data/corpora", run_id="run42")
+    assert lay.sft_path == "/data/corpora/runs/run42/corpus_sft/rows.jsonl"
+    assert lay.manifest_path == "/data/corpora/runs/run42/manifest.json"
+    s3 = RunLayout(root="s3://bucket/prefix/", run_id="r")
+    assert s3.tasks_path == "s3://bucket/prefix/runs/r/tasks/manifest.jsonl"
+
+
+def test_manifest_round_trip_and_budget(tmp_path):
+    lay = RunLayout(root=str(tmp_path), run_id="r1")
+    m = RunManifest(run_id="r1", created_at="2026-06-09T00:00:00Z",
+                    source="test", budget_usd=1.0)
+    m.spend(0.4)
+    assert not m.over_budget
+    m.spend(0.6)
+    assert m.over_budget
+    m.write(lay)
+    m2 = RunManifest.read(lay)
+    assert m2.cost_usd == pytest.approx(1.0)
+    assert m2.budget_usd == 1.0
+
+
+# ---------------------------------------------------------------------
+# THE leak guard (finding D-8)
+# ---------------------------------------------------------------------
+
+
+def test_write_tasks_never_leaks_golden_diff(tmp_path):
+    lay = RunLayout(root=str(tmp_path), run_id="r1")
+    write_tasks(lay, [_task(1)])
+    blob = Path(lay.tasks_path).read_text()
+    assert "SENTINEL_NEVER_LEAK" not in blob
+    assert "secret_fn" not in blob
+    row = json.loads(blob.splitlines()[0])
+    assert row["golden_diff_sha256"]  # provenance preserved as a hash
+    # The restricted full writer DOES carry it (construction side only).
+    write_tasks_full(lay, [_task(1)])
+    assert "SENTINEL_NEVER_LEAK" in Path(lay.tasks_full_path).read_text()
+
+
+# ---------------------------------------------------------------------
+# MinHash dedup
+# ---------------------------------------------------------------------
+
+_TEXT_A = "the quick brown fox jumps over the lazy dog and then runs far away home tonight"
+_TEXT_A2 = "the quick brown fox jumps over the lazy dog and then runs far away home today"
+_TEXT_B = "import numpy as np def main(): return np.zeros(10) print(main()) totally different content here"
+
+
+def test_jaccard_estimate_near_duplicates_high_disjoint_low():
+    sa, sa2, sb = (minhash_signature(t) for t in (_TEXT_A, _TEXT_A2, _TEXT_B))
+    assert jaccard_estimate(sa, sa2) > 0.5
+    assert jaccard_estimate(sa, sb) < 0.2
+    assert jaccard_estimate(sa, sa) == 1.0
+
+
+def test_dedup_keeps_first_and_drops_near_dup():
+    rows = [{"text": _TEXT_A}, {"text": _TEXT_A2}, {"text": _TEXT_B}]
+    kept, stats = dedup(rows, lambda r: r["text"], threshold=0.5)
+    assert [r["text"] for r in kept] == [_TEXT_A, _TEXT_B]
+    assert stats["dropped_within_run"] == 1
+
+
+def test_cross_generation_dedup_via_signature_file():
+    prior_rows = [{"text": _TEXT_A}]
+    buf = io.StringIO()
+    signatures_to_jsonl(prior_rows, lambda r: r["text"], buf)
+    buf.seek(0)
+    prior_sigs = load_signatures(buf)
+
+    rows = [{"text": _TEXT_A2}, {"text": _TEXT_B}]
+    kept, stats = dedup(rows, lambda r: r["text"], threshold=0.5,
+                        prior_signatures=prior_sigs)
+    assert [r["text"] for r in kept] == [_TEXT_B]
+    assert stats["dropped_cross_generation"] == 1
+
+
+def test_find_near_duplicates_pairs():
+    rows = [{"t": _TEXT_A}, {"t": _TEXT_A2}]
+    assert find_near_duplicates(rows, lambda r: r["t"], 0.5) == [(0, 1)]
+
+
+# ---------------------------------------------------------------------
+# build_corpus end-to-end (FakeSandbox + ScriptedPolicy)
+# ---------------------------------------------------------------------
+
+
+def _passing_policy():
+    # Flips both this task's F2P tests green generically: FakeSandbox's
+    # set_outcome takes explicit test names, so the fixture tasks share names
+    # via the same fail_to_pass tuple pattern; we set a superset.
+    outcomes = {f"t/a.py::t{i}": True for i in range(20)}
+    outcomes["t/a.py::keep"] = True
+    return ScriptedPolicy(actions=[ToolCall("set_outcome", {"outcomes": outcomes}), "done"])
+
+
+def _failing_policy():
+    return ScriptedPolicy(actions=["gave up immediately"])
+
+
+def _env():
+    return FeatureDeletionEnv(FakeSandbox(test_outcomes={"t/a.py::keep": True}))
+
+
+def test_build_corpus_end_to_end(tmp_path):
+    tasks = [_task(i) for i in range(6)]
+    lay = RunLayout(root=str(tmp_path), run_id="e2e")
+    manifest = RunManifest(run_id="e2e", created_at="2026-06-09T00:00:00Z", source="fixture")
+
+    out = build_corpus(tasks, _env, _passing_policy, lay, manifest,
+                       holdout_frac=0.34, holdout_seed=7)
+
+    # Holdout exclusion: holdout tasks were never rolled out.
+    assert out.counts["tasks_holdout"] >= 1
+    assert out.counts["rollouts"] == out.counts["tasks_train"]
+    # Full passes routed to SFT (post-dedup near-identical rows collapse —
+    # the fixture tasks produce near-identical messages, which is itself a
+    # realistic dedup scenario).
+    assert out.counts["sft_rows"] >= 1
+    assert out.counts["quarantined"] == 0
+    # Files exist and the SFT corpus never leaks the sentinel.
+    sft_blob = Path(lay.sft_path).read_text()
+    assert "SENTINEL_NEVER_LEAK" not in sft_blob
+    assert Path(lay.card_path).exists()
+    assert Path(lay.holdout_path).exists()
+
+
+def test_build_corpus_quarantines_failures(tmp_path):
+    tasks = [_task(i) for i in range(3)]
+    lay = RunLayout(root=str(tmp_path), run_id="fail")
+    manifest = RunManifest(run_id="fail", created_at="2026-06-09T00:00:00Z", source="fixture")
+    out = build_corpus(tasks, _env, _failing_policy, lay, manifest,
+                       holdout_frac=0.34, holdout_seed=7)
+    assert out.counts["sft_rows"] == 0
+    assert out.counts["quarantined"] == out.counts["rollouts"] > 0
+
+
+def test_build_corpus_budget_stop_marks_partial(tmp_path):
+    tasks = [_task(i) for i in range(6)]
+    lay = RunLayout(root=str(tmp_path), run_id="budget")
+    manifest = RunManifest(run_id="budget", created_at="2026-06-09T00:00:00Z",
+                           source="fixture", budget_usd=0.25)
+    out = build_corpus(tasks, _env, _passing_policy, lay, manifest,
+                       holdout_frac=0.2, holdout_seed=7,
+                       cost_per_rollout_usd=0.1)
+    assert out.status == "partial"
+    assert out.counts["rollouts"] < out.counts["tasks_train"]
+
+
+def test_build_corpus_is_write_once(tmp_path):
+    tasks = [_task(i) for i in range(3)]
+    lay = RunLayout(root=str(tmp_path), run_id="once")
+    m1 = RunManifest(run_id="once", created_at="2026-06-09T00:00:00Z", source="fixture")
+    build_corpus(tasks, _env, _passing_policy, lay, m1, holdout_frac=0.34)
+    m2 = RunManifest(run_id="once", created_at="2026-06-09T00:00:01Z", source="fixture")
+    with pytest.raises(FileExistsError, match="write-once"):
+        build_corpus(tasks, _env, _passing_policy, lay, m2, holdout_frac=0.34)
+
+
+def test_dataset_card_contents(tmp_path):
+    lay = RunLayout(root=str(tmp_path), run_id="card")
+    m = RunManifest(run_id="card", created_at="2026-06-09T00:00:00Z",
+                    source="fixture", counts={"sft_rows": 3})
+    write_dataset_card(lay, m, license_tiers={"REDISTRIBUTABLE": 3},
+                       dedup_stats={"rows_kept": 3})
+    card = Path(lay.card_path).read_text()
+    assert "run `card`" in card
+    assert "sft_rows: 3" in card
+    assert "REDISTRIBUTABLE: 3" in card
+    assert "Decontamination" in card

composer_replication/teacher_replay.py

@@ -4,8 +4,12 @@ This is channel 3 of the integrated trainer: at each step of a frozen agentic
 trace, query N pre-trained external teachers (frontier models from different
 labs) and convert teacher disagreement into preference pairs for DPO loss.
 
-Generalized from spike-001's `replay.py`. Verified economic floor (✅ spike 001):
-$0.98 mean per-trace cost ungated, $0.30/trace projected with VOI gating.
+Generalized from spike-001's `replay.py`. Cost calibration (✅ spike 001,
+relabeled per deepread finding V11): $0.98 mean per-TRACE cost ungated was
+measured on a ~50-state SYNTHETIC trace at N=3 teachers; real Claude Code
+sessions run 125–2,830 tool-use messages (ADR-002), so a full real session is
+~2 orders of magnitude more (~$70–80 flat, before VOI gating). $0.30/trace
+projected with VOI gating, same synthetic basis.
 
 Usage:
     from teacher_replay import replay_trace, extract_dpo_pairs

composer_replication/trainer/kl_in_reward.py

@@ -9,7 +9,9 @@ literature says this is not cosmetic:
 
   * arXiv:2512.21852 ("A Comedy of Estimators") — k1-in-reward improves OOD
     generalization; k3-in-reward can collapse.
-  * verl adopted k1-in-reward as its *only* reverse-KL option.
+  * verl ships k1-in-reward as its default/recommended reverse-KL option
+    (it also supports a k3-family "low_var_kl" — wording corrected per
+    deepread finding V13).
   * TRL issue #4967 tracks the same divergence.
 
 OOD generalization is exactly the "take any model to the next level" axis, so

docs/COMPOSER_RECIPE_MAPPING.md

@@ -22,7 +22,7 @@ The Cursor blog discusses **only three** training innovations explicitly. Everyt
 
 **Cited prior art** (Cursor's footnote 1):
 - **OPSD: Self-Distilled Reasoner — On-Policy Self-Distillation for LLMs** (Zhao et al., 2026, [arXiv:2601.18734](https://arxiv.org/abs/2601.18734), [GitHub: siyan-zhao/OPSD](https://github.com/siyan-zhao/OPSD)). The original on-policy-self-distillation framework: single LLM, teacher conditioned on privileged information (e.g. ground-truth answer), student sees only the question, loss = per-token KL on student's own rollouts.
-- **SDPO: Reinforcement Learning via Self-Distillation** (Hübotter et al., 2026, [arXiv:2601.20802](https://arxiv.org/abs/2601.20802), ICLR 2026 Scaling Post-training Workshop). Generalizes OPSD to RL with rich feedback: *"SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy."* This is **mathematically the same** as Composer's targeted-textual-feedback method. **There is published code.** Comparison table from the SDPO paper:
+- **SDPO: Reinforcement Learning via Self-Distillation** (Hübotter et al., 2026, [arXiv:2601.20802](https://arxiv.org/abs/2601.20802), ICLR 2026 Scaling Post-training Workshop). Generalizes OPSD to RL with rich feedback: *"SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy."* Cursor's blog cites this paper only as **background** ("For more background on this approach see…") — NOT as its mechanism; SDPO's published loss is full-rollout with feedback-in-prefix and an EMA-regularized teacher, while Composer's blog describes a turn-localized hint splice. Closely related, **not verified identical** (deepread finding V1). **There is published code.** Comparison table from the SDPO paper:
 
   | Method | Sampling | Signal | Feedback |
   |---|---|---|---|
@@ -72,7 +72,7 @@ This is **infrastructure, not algorithm**. It only matters at MoE-1T scale; for
 | Composer 2.5 stage | Blog mechanism | Our replication target | v0.0 | v0.1 | v0.2 |
 |---|---|---|---|---|---|
 | **(a)** Continued pretraining on code | Standard pretraining, code-weighted | Skip — start from already-code-tuned `Qwen3-Coder-7B` or `Qwen3-Coder-30B-A3B` | ✗ | ✗ | ✗ |
-| **(b)** Synthetic data at scale | Feature Deletion + 24 other (unnamed) generators | Build 1 generator (Feature Deletion) as OpenEnv-compatible env. Use SWE-bench-lite and SWE-Gym as drop-in alternatives. | ✗ (use SWE-bench-lite only) | ✓ (build Feature Deletion) | scale generator suite |
+| **(b)** Synthetic data at scale | Feature Deletion + an unspecified number of other generators (the blog says only "a range of approaches" — the old "24" was a back-formation from the 25x task multiplier; deepread finding V5) | Build 1 generator (Feature Deletion) as OpenEnv-compatible env. Use SWE-bench-lite and SWE-Gym as drop-in alternatives. | ✗ (use SWE-bench-lite only) | ✓ (build Feature Deletion) | scale generator suite |
 | **(c)** Realistic-environment RL (RLVR) | Async sandboxes, same tool harness as production | TRL `GRPOTrainer` + verifiers + OpenEnv; SWE-bench-lite env in v0.0; build sandboxed code execution env in v0.1 | ✓ baseline | ✓ + DAPO patches | + decentralized rollouts |
 | **(d)** Targeted RL w/ textual feedback (Composer's secret sauce) | Same-model self-distill: insert hint into context → teacher; original → student; on-policy KL at the turn | **Lift the OPSD/SDPO loss directly from `siyan-zhao/OPSD`** (published code, MIT). Generate hints via templates (v0.1) or LLM (v0.2). | ✗ (deferred) | ✓ (this is the Composer-recipe channel) | + learned hint generator |
 | **(e)** Trace-replay multi-teacher distill (NOVEL — our addition) | N/A (not in Composer) | N=3 teachers (Opus 4.7, GPT-5, DeepSeek V4 Pro) replay each step; disagreement → DPO pairs | ✓ (this is the v0.0 novelty bet) | ✓ + VOI gating | + tiered teachers |
@@ -148,7 +148,7 @@ Primary sources for each Composer-2.5 component, post-audit:
 - **Cursor blog** — [Introducing Composer 2.5](https://cursor.com/blog/composer-2-5) (2026)
 - **Cursor blog** — [Composer 2 technical report](https://cursor.com/blog/composer-2-technical-report) (predecessor; named the "Anyrun" environment per subagent — verify if needed)
 - **OPSD paper** — Zhao et al., *Self-Distilled Reasoner: On-Policy Self-Distillation for LLMs*, [arXiv:2601.18734](https://arxiv.org/abs/2601.18734), code at [siyan-zhao/OPSD](https://github.com/siyan-zhao/OPSD). MIT.
-- **SDPO paper** — Hübotter et al., *Reinforcement Learning via Self-Distillation*, [arXiv:2601.20802](https://arxiv.org/abs/2601.20802), ICLR 2026 Scaling Post-training Workshop. The direct formalization of Composer's hint-distill.
+- **SDPO paper** — Hübotter et al., *Reinforcement Learning via Self-Distillation*, [arXiv:2601.20802](https://arxiv.org/abs/2601.20802), ICLR 2026 Scaling Post-training Workshop. The closest published formalization; cited by Cursor only as background (deepread finding V1).
 - **Self-Distillation continual-learning** — [arXiv:2601.19897](https://arxiv.org/abs/2601.19897). Cited by Cursor; less directly relevant.
 - **Moonshot Kimi K2.5** — base model, [HF model card](https://huggingface.co/moonshotai/Kimi-K2-Thinking).
 

docs/adrs/ADR-016-stage0-dataset-pipeline.md

@@ -0,0 +1,119 @@
+---
+status: accepted
+date: 2026-06-09
+deciders: [Codeseys, ARIA]
+---
+
+# ADR-016: Stage-0 dataset-generation pipeline — SWE-smith engine + rollout harness + ingest gates + single contract
+
+## Context and Problem Statement
+
+The user asked to "architect and build a pipeline that builds out a dataset like
+the Composer 2.5 blog mentions, with our vision enhancements — point to an
+open-source repo and use that to build the dataset, or use traces or other
+datasets and enhance them."
+
+Before building, a full-source critical review re-read every foundational paper
+and blog (8 source clusters, `research/deepread/01-08`), ground-mapped the repo
+(`00`), ran adversarial fidelity + design critics, and independently VERIFIED
+every finding (`12` — 0 refuted). The verified verdict: the envisioned pipeline
+had four structural breaks (seed-trace/oracle disjointness; no rollout harness —
+the SFT corpus had NO producer; an uncomputable divergence gate; no
+`Sandbox.fork()`), several missing controls (zero benchmark decontamination,
+no secrets gate, a `golden_diff` serialization leak, two unreconciled S3
+contracts, no cross-generation dedup), and a buy-vs-build inversion (the
+planned image-builder duplicates `pip install swesmith`, whose PR-Mirror
+strategy IS this repo's gold-patch-reversion mechanic and is validated best-of-
+five by SWE-smith's own ablation, Table 5 of arXiv:2504.21798).
+
+## Decision
+
+Build **Stage 0 local-first** (architecture: `research/deepread/13-synthesis-architecture.md`):
+
+1. **SWE-smith is the synthesis engine** for "point at a repo" (`[swesmith]`
+   extra; `datagen/swesmith_adapter.py` bridges its instances into
+   `FeatureDeletionTask`, handling the patch-semantics INVERSION — SWE-smith's
+   patch introduces the bug, so `golden_diff` = `reverse_unified_diff(patch)`).
+   `SweBenchAdapter` remains the bridge for SWE-bench-shaped substrates.
+2. **Ingest gates before anything else** (`datagen/repo_gate.py`): SPDX-ish
+   license detection → three tiers (REDISTRIBUTABLE / TRAINABLE_ONLY /
+   EXCLUDED, fail-closed) + **benchmark decontamination** against the
+   SWE-bench-family eval-repo list (hard fail).
+3. **The rollout harness is the corpus producer** (`datagen/rollout_harness.py`):
+   `collect_trajectory(env, task, policy)` runs a pluggable policy
+   (ScriptedPolicy for tests; OpenRouterPolicy stub; mini-swe-agent/SWE-agent
+   adoption is the documented upgrade) through `FeatureDeletionEnv` to
+   `_grade()`. Its env-grounded trajectories are ALSO the tree-of-work's seed
+   nodes — fixing the seed/oracle disjointness as a byproduct. `admit()` routes
+   typed signal: clean full pass → SFT; clean near-miss → DPO candidate;
+   guard-broken/hacked → quarantine (never raw negative gradient).
+4. **One canonical trajectory IR** (`datagen/trajectory.py`): `ToolCall` (whose
+   `canonical_form()` is the v1 divergence-gate action algebra, replacing the
+   whitespace stub), `CanonicalTrajectory`, adapters from Claude Code traces
+   (explicitly UNGRADED — demoted to flat/SFT uses), and `to_policy_row()` —
+   the ONE policy-visible serializer, unit-tested to never emit
+   `golden_diff`/`deleted_symbols` (sentinel test).
+5. **One reconciled dataset contract** (`pipeline/s3_contract.py`, supersedes
+   design-F1's and design-F2's divergent layouts): `runs/<id>/{tasks,
+   tasks_full(RESTRICTED), traj, corpus_sft, corpus_dpo, holdout, quarantine}`
+   + `RunManifest` (counts, cost, budget, `parent_run_id` lineage, status) +
+   dataset card. Policy-safe task rows carry `golden_diff_sha256`, never the
+   diff. DiLoCo rendezvous and `wm_tuples/` are deliberately OUT (separate
+   concern; ablation-gated respectively).
+6. **Cross-generation dedup** (`pipeline/dedup.py`): stable-hash MinHash over
+   word 5-shingles; a run can dedup against the prior generation's signature
+   file (flywheel-collapse mitigation). datasketch/LSH is the upgrade path.
+7. **The local stage-driver** (`pipeline/build_corpus.py`): holdout-split FIRST
+   (held-out tasks never rolled out), rollouts under a budget ceiling
+   (partial-marking), typed routing, dedup, write-once-per-run idempotency.
+
+## Fidelity corrections shipped with this ADR (deepread findings, all verified)
+
+- **V1:** "SDPO is mathematically the same as Composer's mechanism" corrected
+  in `opsd.py` + `COMPOSER_RECIPE_MAPPING.md` — Cursor cites SDPO/OPSD as
+  *background*; our channel is a third, blog-inspired design (turn-localized
+  hint splice, live stop-grad teacher, no EMA).
+- **V5:** fabricated numbers struck/tagged: 69.3%/Terminal-Bench parity (no
+  primary source), "24 other generators" (back-formed), "85% post-training
+  compute" (community speculation) — `research/01`, mapping doc, `research/06`,
+  `research/09`.
+- **V7:** Streaming DiLoCo citation fixed (`diloco/__init__.py`): 2501.18512 =
+  Douillard et al.; Eager Updates = Kale et al. 2502.12996.
+- **V11:** `teacher_replay.py` cost docstring relabeled ($0.98 = 50-state
+  synthetic trace; real sessions ~2 OOM more).
+- **V13:** `kl_in_reward.py` "verl's only reverse-KL option" → default/
+  recommended (verl also ships a k3-family option).
+
+## What is deliberately NOT in Stage 0
+
+- AWS orchestration (Glue/EMR/Batch/Bedrock-batch/Step Functions) — Stage 4,
+  only after local runs are routine (finding D-9).
+- Tree depth>1 — gated on a `Sandbox.fork()` spike + a measured divergence-gate
+  firing rate (findings D-3/D-4). Depth-1 multi-candidate rollouts need no fork.
+- World-model `wm_tuples/` emission — gated on the P4 ablation being scheduled
+  (finding D-14; CWM evidence is mid-training, not RL-time aux head — V6).
+- Secrets/PII scrub at trace ingest (finding V9) — REQUIRED before any raw
+  Claude Code session is uploaded to shared storage; tracked as the next
+  pipeline item. Local-only runs are unaffected.
+
+## Acceptance gate
+
+- [x] `repo_gate`: 53 tests (license tiers, decontamination, gate verdicts).
+- [x] `swesmith_adapter`: 18 tests (patch INVERSION semantics, reverse round-trip,
+      strategy provenance, image conventions).
+- [x] `trajectory` + `rollout_harness`: 13 tests (IR round-trips, SENTINEL leak
+      guard, env-grounded episode to grade 1.0 / guard-broken / near-miss,
+      admission routing).
+- [x] `pipeline`: 12 tests (layout, manifest+budget, leak guard at the writer,
+      MinHash within-run + cross-generation, build_corpus e2e with holdout
+      exclusion + budget stop + write-once).
+- [x] Full suite green: 511 passed / 66 skipped.
+- [ ] Live swesmith synthesis on a real pointed-at repo (needs Docker+Linux) —
+      the documented `[~]` gate, same shape as ADR-010's Docker e2e.
+
+## More Information
+
+- `research/deepread/13-synthesis-architecture.md` — the architecture this implements.
+- `research/deepread/12-verified-findings.md` — the verified finding ledger (V1–V15).
+- `research/deepread/02-swe-task-synthesis.md` — the SWE-smith/R2E-Gym/SWE-Gym deep-read.
+- ADR-010 (the substrate-inversion base this extends), ADR-002 (trace source).

pyproject.toml

@@ -83,6 +83,14 @@ aws = [
     "boto3>=1.34",
     "sagemaker>=2.200,<3",
 ]
+# SWE-smith task-synthesis engine (deepread finding V4 buy-vs-build verdict):
+# the swesmith toolkit builds env images from arbitrary GitHub repos and
+# synthesizes bugs (PR Mirror = this repo's gold-patch-reversion mechanic).
+# LIVE synthesis needs Docker on Linux (the toolkit does not support macOS/
+# Windows officially); the SwesmithAdapter itself needs nothing beyond core.
+swesmith = [
+    "swesmith>=0.1",
+]
 # Replaysim dataset normalization (per ADR-004)
 #
 # NOTE: data-juicer is intentionally NOT pinned as an extra. The package

research/01-composer-2.5.md

@@ -11,7 +11,7 @@
 > The targeted-textual-feedback method is correctly described, but this file does **not** cite the three self-distillation papers Cursor cites in footnote 1 (OPSD `arXiv:2601.18734`, SDPO `arXiv:2601.20802`, Self-Distillation Continual Learning `arXiv:2601.19897`). The mapping document does.
 
 ## Overview
-Cursor's Composer 2.5 is an advanced agentic coding model that powers the Cursor IDE. Released in mid-May 2026, it represents a massive leap in agentic capabilities, particularly for long-running, multi-file software engineering tasks. While the base weights are Moonshot AI's open-source **Kimi K2.5** model, roughly 85% of the total compute budget for Composer 2.5 was spent on Cursor's proprietary post-training and Reinforcement Learning (RL) pipeline. 
+Cursor's Composer 2.5 is an advanced agentic coding model that powers the Cursor IDE. Released in mid-May 2026, it represents a massive leap in agentic capabilities, particularly for long-running, multi-file software engineering tasks. While the base weights are Moonshot AI's open-source **Kimi K2.5** model, a large share of the compute budget went to Cursor's proprietary post-training/RL pipeline (the widely-circulated "85%" figure is community speculation, in NO primary source — deepread finding V5). 
 
 The resulting model is highly optimized for the exact constraints and tools of the Cursor environment (file edits, terminal usage, LSP interaction). Composer 2.5 is praised for having fewer "false-start" tool calls, avoiding prompt-baiting, and demonstrating a much calmer, more effective collaboration loop than its predecessors.
 
@@ -60,8 +60,8 @@ During post-training, Cursor employs **Sharded Muon** and **Dual Mesh HSDP (Hybr
 ## Performance Characteristics
 Cursor claims Composer 2.5 achieves a Pareto-optimal tradeoff between intelligence and inference cost compared to frontier models (Opus 4.5/4.6, GPT-5.4/5.5).
 
-*   **Intelligence Improvements**: On Cursor's internal *CursorBench* (which tests sweeping, multi-file edits with ambiguous prompts), Composer 2.5 scored 69.3% (or ~61-63% depending on the specific benchmark version cited), a massive jump from Composer 1.5's ~44% and Composer 2's ~52%.
-*   **Frontier Parity**: On public agentic benchmarks like *Terminal-Bench 2.0*, it hit 69.3%. On *SWE-bench Multilingual*, it achieved parity with or slightly surpassed OpenAI's GPT-5.5.
+*   **Intelligence Improvements**: On Cursor's internal *CursorBench* (which tests sweeping, multi-file edits with ambiguous prompts), Composer 2.5's score is NOT in any primary source (the circulating 69.3% figure appears in neither the 2.5 blog nor the Composer 2 techreport — deepread finding V5; the techreport's Table 1 gives Composer 2 = 61.3 CursorBench). Treat all 2.5 benchmark numbers as unverified.
+*   **Frontier Parity**: Claims of Terminal-Bench 2.0 / SWE-bench Multilingual parity circulate in secondary commentary only; neither primary source contains benchmark numbers for 2.5 (deepread finding V5).
 *   **Cost Efficiency**: 
     *   Standard Tier: $0.50 per 1M input / $2.50 per 1M output tokens.
     *   Fast Tier: $3.00 per 1M input / $15.00 per 1M output tokens.

research/06-feature-deletion-datagen.md

@@ -327,7 +327,7 @@ Feature-Deletion is **embarrassingly parallel and CPU-bound** — no GPU in the
 
 1. **Deletion-target selection heuristic** — blog silent (`research/09` §1 "NO CHANGE"). We propose coverage-selectivity (§5 Path B); Cursor's actual heuristic is unknown.
 2. **Deleter model vs. program** — blog implies an agent deletes ("asked to delete code… such that the codebase remains functional"); we default to *programmatic* deletion (cheaper, deterministic, no second model). An LLM-deleter is a v0.2 escalation.
-3. **The other ~24 generators** — Feature Deletion is "one synthetic approach… a range of approaches"; the rest are unnamed. Out of scope here; this brief delivers the one named generator.
+3. **The other generators (count UNKNOWN)** — Feature Deletion is "one synthetic approach… a range of approaches"; the rest are unnamed and uncounted (the old "~24" was a back-formation from the 25x task multiplier — deepread finding V5). Out of scope here; this brief delivers the one named generator.
 4. **"Agentic monitoring tools" internals** — unspecified; our §3c monitor is a best-effort programmatic stand-in.
 5. **Composer2.pdf (arXiv:2603.24477)** — flagged by `research/09` action-item #1 as the likely home of data-mix % and generator inventory; **not yet extracted**. Recommend a follow-up pull before scaling the generator suite.
 

research/09-composer-blog-delta-2026.md

@@ -20,7 +20,7 @@ The **2.5 blog body is byte-for-byte unchanged** from what the mapping doc captu
 
 **DELTAS (not in / under-stated in COMPOSER_RECIPE_MAPPING.md):**
 
-- **[DELTA — new emphasis]** The phrase *"we both **select for** and **create** harder tasks **dynamically throughout the run**"* is a **dynamic curriculum / online task-selection** signal. The mapping doc captured "Feature Deletion + 24 unnamed generators" but did **not** flag that task difficulty is filtered *online* (the model "begins to get most training problems correct," so hard tasks are up-weighted live). This is a data-*mix*/curriculum detail with direct replication impact: our generator suite needs a difficulty filter / pass-rate gate, not just a static task bank.
+- **[DELTA — new emphasis]** The phrase *"we both **select for** and **create** harder tasks **dynamically throughout the run**"* is a **dynamic curriculum / online task-selection** signal. The mapping doc captured "Feature Deletion + other unnamed generators" (its old "24" count was a back-formation — deepread finding V5) but did **not** flag that task difficulty is filtered *online* (the model "begins to get most training problems correct," so hard tasks are up-weighted live). This is a data-*mix*/curriculum detail with direct replication impact: our generator suite needs a difficulty filter / pass-rate gate, not just a static task bank.
 - **[DELTA — new authoritative source for CPT data mix]** The Composer 2 technical-report blog states the CPT data mix explicitly: *"continued pretraining on a data mix that **emphasizes code** to deepen the base model's coding knowledge"* and *"We find that **reducing pretraining loss improves downstream RL performance**, with better base knowledge reliably translating into a better agent."* The mapping doc marked "continued pretraining on heavily code-weighted data" as `[BLOG-VERIFIED]` from the 2.5 Muon section — but the **causal claim (CPT loss ↓ ⇒ RL performance ↑)** is new and is the stated *justification* for doing CPT at all. Relevant to our "skip CPT, start from Qwen3-Coder" decision: Cursor's own evidence says base-knowledge quality gates RL ceiling, which strengthens the case for starting from an already-code-tuned base.
 - **[DELTA — new artifact]** There is now a **full Composer 2 arXiv technical report: [arXiv:2603.24477](https://arxiv.org/abs/2603.24477)** and a downloadable PDF at **`https://cursor.com/resources/Composer2.pdf`** (authored by Sasha Rush et al.). The report explicitly *"covers... ablations on the training recipe, our approach to agent behavior shaping, and the design of our evaluation suite."* The mapping doc cited only the blog stub and never the arXiv ID/PDF. **This PDF is the most likely place to resolve the data-mix weighting %, the RL algorithm name, and the hint-generation mechanism — none of which are in either blog.** → Recommend a dedicated follow-up extraction of Composer2.pdf.
 - **[CONFIRM — "Anyrun"]** Mapping doc flagged "Anyrun" as possibly not Cursor-sourced. **Confirmed real:** the Composer 2 report blog says *"**Anyrun**, our internal compute platform for running hundreds of thousands of sandboxed coding environments."* It is a Composer-**2** artifact (carried into 2.5), correctly attributed. Resolves the mapping doc's open flag.

research/notes/230406767-raft-reward-ranked-finetuning-for-generative-foundation-model-alignmen.md

@@ -0,0 +1,224 @@
+---
+title: '[2304.06767] RAFT: Reward rAnked FineTuning for Generative Foundation Model
+  Alignment'
+id: 230406767-raft-reward-ranked-finetuning-for-generative-foundation-model-alignmen
+tags:
+- deepread
+created: '2026-06-10T00:31:18.566124Z'
+source: https://arxiv.org/abs/2304.06767
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:31:18.565918Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2304.06767] RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
+Computer Science > Machine Learning
+arXiv:2304.06767
+(cs)
+[Submitted on 13 Apr 2023 (
+v1
+), last revised 1 Dec 2023 (this version, v4)]
+Title:
+RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
+Authors:
+Hanze Dong
+,
+Wei Xiong
+,
+Deepanshu Goyal
+,
+Yihan Zhang
+,
+Winnie Chow
+,
+Rui Pan
+,
+Shizhe Diao
+,
+Jipeng Zhang
+,
+Kashun Shum
+,
+Tong Zhang
+View a PDF of the paper titled RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment, by Hanze Dong and 9 other authors
+View PDF
+HTML (experimental)
+Abstract:
+Generative foundation models are susceptible to implicit biases that can arise from extensive unsupervised training data. Such biases can produce suboptimal samples, skewed outcomes, and unfairness, with potentially serious consequences. Consequently, aligning these models with human ethics and preferences is an essential step toward ensuring their responsible and effective deployment in real-world applications. Prior research has primarily employed Reinforcement Learning from Human Feedback (RLHF) to address this problem, where generative models are fine-tuned with RL algorithms guided by a human-feedback-informed reward model. However, the inefficiencies and instabilities associated with RL algorithms frequently present substantial obstacles to the successful alignment, necessitating the development of a more robust and streamlined approach. To this end, we introduce a new framework, Reward rAnked FineTuning (RAFT), designed to align generative models effectively. Utilizing a reward model and a sufficient number of samples, our approach selects the high-quality samples, discarding those that exhibit undesired behavior, and subsequently enhancing the model by fine-tuning on these filtered samples. Our studies show that RAFT can effectively improve the model performance in both reward learning and other automated metrics in both large language models and diffusion models.
+Comments:
+29 pages, 12 figures, Published in Transactions on Machine Learning Research (TMLR)
+Subjects:
+Machine Learning (cs.LG)
+; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)
+Cite as:
+arXiv:2304.06767
+[cs.LG]
+(or
+arXiv:2304.06767v4
+[cs.LG]
+for this version)
+https://doi.org/10.48550/arXiv.2304.06767
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Hanze Dong [
+view email
+]
+[v1]
+Thu, 13 Apr 2023 18:22:40 UTC (62,967 KB)
+[v2]
+Thu, 25 May 2023 06:27:31 UTC (42,022 KB)
+[v3]
+Wed, 30 Aug 2023 01:25:29 UTC (33,955 KB)
+[v4]
+Fri, 1 Dec 2023 14:28:06 UTC (34,049 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment, by Hanze Dong and 9 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.LG
+< prev
+|
+next >
+new
+|
+recent
+|
+2023-04
+Change to browse by:
+cs
+cs.AI
+cs.CL
+cs.CV
+stat
+stat.ML
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+Links to Code Toggle
+Papers with Code
+(
+What is Papers with Code?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+IArxiv recommender toggle
+IArxiv Recommender
+(
+What is IArxiv?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/230510601-tree-of-thoughts-deliberate-problem-solving-with-large-language-models-2.md

@@ -0,0 +1,2735 @@
+---
+title: '[2305.10601] Tree of Thoughts: Deliberate Problem Solving with Large Language
+  Models'
+id: 230510601-tree-of-thoughts-deliberate-problem-solving-with-large-language-models-2
+tags:
+- deepread
+created: '2026-06-10T00:41:12.876142Z'
+source: https://ar5iv.labs.arxiv.org/html/2305.10601
+source_domain: ar5iv.labs.arxiv.org
+fetched_at: '2026-06-10T00:41:12.875985Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2305.10601] Tree of Thoughts: Deliberate Problem Solving with Large Language Models
+Tree of Thoughts: Deliberate Problem Solving
+with Large Language Models
+Shunyu Yao
+Princeton University
+Dian Yu
+Google DeepMind
+Jeffrey Zhao
+Google DeepMind
+Izhak Shafran
+Google DeepMind
+Thomas L. Griffiths
+Princeton University
+Yuan Cao
+Google DeepMind
+Karthik Narasimhan
+Princeton University
+Abstract
+Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role.
+To surmount these challenges, we introduce a new framework for language model inference, “Tree of Thoughts” (ToT), which generalizes over the popular “Chain of Thought” approach to prompting language models, and enables exploration over coherent units of text (“thoughts”) that serve as intermediate steps toward problem solving.
+ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.
+Our experiments show that ToT significantly enhances language models’ problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords.
+For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%. Code repo with all prompts:
+https://github.com/princeton-nlp/tree-of-thought-llm
+.
+1
+Introduction
+Originally designed to generate text, scaled-up versions of language models (LMs) such as GPT
+[
+25
+,
+26
+,
+1
+,
+23
+]
+and PaLM
+[
+5
+]
+have been shown to be increasingly capable of performing an ever wider range of tasks requiring mathematical, symbolic, commonsense, and knowledge reasoning. It is perhaps surprising that underlying all this progress is still the original autoregressive mechanism for generating text, which makes token-level decisions one by one and in a left-to-right fashion.
+Is such a simple mechanism sufficient for a LM to be built toward a general problem solver?
+If not, what problems would challenge the current paradigm, and what should be alternative mechanisms?
+The literature on human cognition provides some clues to answer these questions.
+Research on “dual process” models suggests that people have two modes in which they engage with decisions – a fast, automatic, unconscious mode (“System 1”) and a slow, deliberate, conscious mode (“System 2”)
+[
+30
+,
+31
+,
+16
+,
+15
+]
+.
+These two modes have previously been connected to a variety of mathematical models used in machine learning. For example, research on reinforcement learning in humans and other animals has explored the circumstances under which they engage in associative “model free” learning or more deliberative “model based” planning
+[
+7
+]
+.
+The simple associative token-level choices of LMs are also reminiscent of “System 1”, and thus might benefit from augmentation by a more deliberate “System 2” planning process that (1) maintains and explores diverse alternatives for current choices instead of just picking one, and (2) evaluates its current status and actively looks ahead or backtracks to make more global decisions.
+To design such a planning process, we return to the origins of artificial intelligence (and cognitive science), drawing inspiration from the planning processes explored by Newell, Shaw, and Simon starting in the 1950s
+[
+21
+,
+22
+]
+. Newell and colleagues characterized
+problem solving
+[
+21
+]
+as search through a combinatorial problem space, represented as a tree. We thus propose the Tree of Thoughts (ToT) framework for general problem solving with language models. As Figure
+1
+illustrates, while existing methods (detailed below) sample continuous language sequences for problem solving, ToT actively maintains a tree of thoughts, where each
+thought
+is a coherent language sequence that serves as an intermediate step toward problem solving (Table
+1
+). Such a high-level semantic unit allows the LM to self-evaluate the progress different intermediate thoughts make towards solving the problem through a deliberate reasoning process that is also instantiated in language (Figures
+2
+,
+4
+,
+6
+). This implementation of search heuristics via LM self-evaluation and deliberation is novel, as previous search heuristics are either programmed or learned. Finally, we combine this language-based capability to generate and evaluate diverse thoughts with search algorithms, such as breadth-first search (BFS) or depth-first search (DFS), which allow systematic exploration of the tree of thoughts with lookahead and backtracking.
+Empirically, we propose three new problems that challenge existing LM inference methods even with the state-of-the-art language model, GPT-4
+[
+23
+]
+: Game of 24, Creative Writing, and Crosswords (Table
+1
+).
+These tasks require deductive, mathematical, commonsense, lexical reasoning abilities, and a way to incorporate systematic planning or search.
+We show ToT obtains superior results on all three tasks by being general and flexible enough to support different levels of thoughts, different ways to generate and evaluate thoughts, and different search algorithms that adapt to the nature of different problems. We also analyze how such choices affect model performances via systematic ablations and discuss future directions to better train and use LMs.
+Figure 1:
+Schematic illustrating various approaches to problem solving with LLMs. Each rectangle box represents a
+thought
+, which is a coherent language sequence that serves as an intermediate step toward problem solving. See concrete examples of how thoughts are generated, evaluated, and searched in Figures
+2
+,
+4
+,
+6
+.
+2
+Background
+We first formalize some existing methods that use large language models for problem-solving, which our approach is inspired by and later compared with.
+We use
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+to denote a pre-trained LM with parameters
+θ
+𝜃
+\theta
+, and
+lowercase letters
+x
+,
+y
+,
+z
+,
+s
+,
+⋯
+𝑥
+𝑦
+𝑧
+𝑠
+⋯
+x,y,z,s,\cdots
+to denote a language sequence
+, i.e.
+x
+=
+(
+x
+​
+[
+1
+]
+,
+⋯
+,
+x
+​
+[
+n
+]
+)
+𝑥
+𝑥
+delimited-[]
+1
+⋯
+𝑥
+delimited-[]
+𝑛
+x=(x[1],\cdots,x[n])
+where each
+x
+​
+[
+i
+]
+𝑥
+delimited-[]
+𝑖
+x[i]
+is a token, so that
+p
+θ
+​
+(
+x
+)
+=
+∏
+i
+=
+1
+n
+p
+θ
+​
+(
+x
+​
+[
+i
+]
+|
+x
+​
+[
+1
+​
+…
+​
+i
+]
+)
+subscript
+𝑝
+𝜃
+𝑥
+superscript
+subscript
+product
+𝑖
+1
+𝑛
+subscript
+𝑝
+𝜃
+conditional
+𝑥
+delimited-[]
+𝑖
+𝑥
+delimited-[]
+1
+…
+𝑖
+p_{\theta}(x)=\prod_{i=1}^{n}p_{\theta}(x[i]|x[1...i])
+. We use uppercase letters
+S
+,
+⋯
+𝑆
+⋯
+S,\cdots
+to denote a collection of language sequences.
+Input-output (IO) prompting
+is the most common way to turn a problem input
+x
+𝑥
+x
+into output
+y
+𝑦
+y
+with LM:
+y
+∼
+p
+θ
+​
+(
+y
+|
+prompt
+I
+​
+O
+​
+(
+x
+)
+)
+similar-to
+𝑦
+subscript
+𝑝
+𝜃
+conditional
+𝑦
+subscript
+prompt
+𝐼
+𝑂
+𝑥
+y\sim p_{\theta}(y|\texttt{prompt}_{{IO}}(x))
+, where
+prompt
+I
+​
+O
+​
+(
+x
+)
+subscript
+prompt
+𝐼
+𝑂
+𝑥
+\texttt{prompt}_{IO}(x)
+wraps input
+x
+𝑥
+x
+with task instructions and/or few-shot input-output examples. For simplicity, let us denote
+p
+θ
+prompt
+​
+(
+output
+∣
+input
+)
+=
+p
+θ
+​
+(
+output
+∣
+prompt
+​
+(
+input
+)
+)
+superscript
+subscript
+𝑝
+𝜃
+prompt
+conditional
+output
+input
+subscript
+𝑝
+𝜃
+conditional
+output
+prompt
+input
+p_{\theta}^{{\rm prompt}}(\texttt{output}\mid\texttt{input})=p_{\theta}(\texttt{output}\mid\texttt{prompt}(\texttt{input}))
+, so that IO prompting can be formulated as
+y
+∼
+p
+θ
+I
+​
+O
+​
+(
+y
+|
+x
+)
+similar-to
+𝑦
+superscript
+subscript
+𝑝
+𝜃
+𝐼
+𝑂
+conditional
+𝑦
+𝑥
+y\sim p_{\theta}^{IO}(y|x)
+.
+Chain-of-thought (CoT) prompting
+[
+38
+]
+was proposed to address cases where the mapping of input
+x
+𝑥
+x
+to output
+y
+𝑦
+y
+is non-trivial (e.g. when
+x
+𝑥
+x
+is a math question and
+y
+𝑦
+y
+is the final numerical answer). The key idea is to introduce a chain of
+thoughts
+z
+1
+,
+⋯
+,
+z
+n
+subscript
+𝑧
+1
+⋯
+subscript
+𝑧
+𝑛
+z_{1},\cdots,z_{n}
+to bridge
+x
+𝑥
+x
+and
+y
+𝑦
+y
+, where each
+z
+i
+subscript
+𝑧
+𝑖
+z_{i}
+is a coherent language sequence that serves as a meaningful intermediate step toward problem solving (e.g.
+z
+i
+subscript
+𝑧
+𝑖
+z_{i}
+could be an intermediate equation for math QA). To solve problems with CoT, each thought
+z
+i
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+z
+i
+∣
+x
+,
+z
+1
+​
+⋯
+​
+i
+−
+1
+)
+similar-to
+subscript
+𝑧
+𝑖
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+subscript
+𝑧
+𝑖
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑖
+1
+z_{i}\sim p_{\theta}^{CoT}(z_{i}\mid x,z_{1\cdots i-1})
+is sampled sequentially, then the output
+y
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+y
+|
+x
+,
+z
+1
+​
+⋯
+​
+n
+)
+similar-to
+𝑦
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+𝑦
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑛
+y\sim p_{\theta}^{CoT}(y|x,z_{1\cdots n})
+. In practice,
+[
+z
+1
+​
+⋯
+​
+n
+,
+y
+]
+��
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+z
+1
+​
+⋯
+​
+n
+,
+y
+|
+x
+)
+similar-to
+subscript
+𝑧
+1
+⋯
+𝑛
+𝑦
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+subscript
+𝑧
+1
+⋯
+𝑛
+conditional
+𝑦
+𝑥
+[z_{1\cdots n},y]\sim p_{\theta}^{CoT}(z_{1\cdots n},y|x)
+is sampled as a continuous language sequence, and the
+decomposition
+of thoughts (e.g. is each
+z
+i
+subscript
+𝑧
+𝑖
+z_{i}
+a phrase, a sentence, or a paragraph) is left ambiguous.
+Self-consistency with CoT (CoT-SC)
+[
+36
+]
+is an ensemble approach that samples
+k
+𝑘
+k
+i.i.d. chains of thought:
+[
+z
+1
+​
+⋯
+​
+n
+(
+i
+)
+,
+y
+(
+i
+)
+]
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+z
+1
+​
+⋯
+​
+n
+,
+y
+|
+x
+)
+​
+(
+i
+=
+1
+​
+⋯
+​
+k
+)
+similar-to
+subscript
+superscript
+𝑧
+𝑖
+1
+⋯
+𝑛
+superscript
+𝑦
+𝑖
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+subscript
+𝑧
+1
+⋯
+𝑛
+conditional
+𝑦
+𝑥
+𝑖
+1
+⋯
+𝑘
+[z^{(i)}_{1\cdots n},y^{(i)}]\sim p_{\theta}^{CoT}(z_{1\cdots n},y|x)\ (i=1\cdots k)
+, then returns the most frequent output:
+arg
+⁡
+max
+y
+⁡
+#
+​
+{
+i
+∣
+y
+(
+i
+)
+=
+y
+}
+subscript
+𝑦
+#
+conditional-set
+𝑖
+superscript
+𝑦
+𝑖
+𝑦
+\arg\max_{y}\#\{i\mid y^{(i)}=y\}
+. CoT-SC improves upon CoT, because there are generally different thought processes for the same problem (e.g. different ways to prove the same theorem), and the output decision can be more faithful by exploring a richer set of thoughts. However, within each chain there is no local exploration of different thought steps, and the “most frequent” heuristic only applies when the output space is limited (e.g. multi-choice QA).
+3
+Tree of Thoughts: Deliberate Problem Solving with LM
+A genuine problem-solving process involves the repeated use of available information to initiate exploration, which discloses, in turn, more information until a way to attain the solution is finally discovered.——
+Newell et al. [
+21
+]
+Research on human problem-solving suggests that people search through a combinatorial problem-space – a tree where the nodes represent partial solutions, and the branches correspond to operators that modify them
+[
+21
+,
+22
+]
+. Which branch to take is determined by heuristics that help to navigate the problem-space and guide the problem-solver towards a solution. This perspective highlights two key shortcomings of existing approaches that use LMs to solve general problems: 1) Locally, they do not explore
+different
+continuations within a thought process – the branches of the tree. 2) Globally, they do not incorporate any type of planning, lookahead, or backtracking to help evaluate these different options – the kind of heuristic-guided search that seems characteristic of human problem-solving.
+To address these shortcomings, we introduce
+Tree of Thoughts (ToT)
+, a paradigm that allows LMs to explore multiple reasoning paths over thoughts (Figure
+1
+(c)). ToT frames any problem as a search over a tree, where each node is a
+state
+s
+=
+[
+x
+,
+z
+1
+​
+⋯
+​
+i
+]
+𝑠
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑖
+s=[x,z_{1\cdots i}]
+representing a partial solution with the input and the sequence of thoughts so far. A specific instantiation of ToT involves answering four questions: 1. How to
+decompose
+the intermediate process into thought steps; 2. How to
+generate
+potential thoughts from each state; 3. How to heuristically
+evaluate
+states; 4. What
+search
+algorithm to use.
+1. Thought decomposition.
+While CoT samples thoughts coherently without explicit decomposition, ToT leverages problem properties to design and decompose intermediate thought steps. As Table
+1
+shows, depending on different problems, a thought could be a couple of words (Crosswords), a line of equation (Game of 24), or a whole paragraph of writing plan (Creative Writing). In general, a thought should be “small” enough so that LMs can generate promising and diverse samples (e.g. generating a whole book is usually too “big” to be coherent), yet “big” enough so that LMs can evaluate its prospect toward problem solving (e.g. generating one token is usually too “small” to evaluate).
+2. Thought generator
+G
+​
+(
+p
+θ
+,
+s
+,
+k
+)
+𝐺
+subscript
+𝑝
+𝜃
+𝑠
+𝑘
+G(p_{\theta},s,k)
+.
+Given a tree state
+s
+=
+[
+x
+,
+z
+1
+​
+⋯
+​
+i
+]
+𝑠
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑖
+s=[x,z_{1\cdots i}]
+, we consider two strategies to generate
+k
+𝑘
+k
+candidates for the next thought step:
+(a)
+Sample
+i.i.d. thoughts from a CoT prompt (Creative Writing, Figure
+4
+):
+z
+(
+j
+)
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+z
+i
++
+1
+|
+s
+)
+=
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+z
+i
++
+1
+|
+x
+,
+z
+1
+​
+⋯
+​
+i
+)
+​
+(
+j
+=
+1
+​
+⋯
+​
+k
+)
+similar-to
+superscript
+𝑧
+𝑗
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+subscript
+𝑧
+𝑖
+1
+𝑠
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+subscript
+𝑧
+𝑖
+1
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑖
+𝑗
+1
+⋯
+𝑘
+z^{(j)}\sim p_{\theta}^{CoT}(z_{i+1}|s)=p_{\theta}^{CoT}(z_{i+1}|x,z_{1\cdots i})\ (j=1\cdots k)
+. This works better when the thought space is rich (e.g. each thought is a paragraph), and i.i.d. samples lead to diversity;
+(b)
+Propose
+thoughts sequentially using a “propose prompt” (Game of 24, Figure
+2
+; Crosswords, Figure
+6
+):
+[
+z
+(
+1
+)
+,
+⋯
+,
+z
+(
+k
+)
+]
+∼
+p
+θ
+p
+​
+r
+​
+o
+​
+p
+​
+o
+​
+s
+​
+e
+​
+(
+z
+i
++
+1
+(
+1
+​
+⋯
+​
+k
+)
+∣
+s
+)
+similar-to
+superscript
+𝑧
+1
+⋯
+superscript
+𝑧
+𝑘
+superscript
+subscript
+𝑝
+𝜃
+𝑝
+𝑟
+𝑜
+𝑝
+𝑜
+𝑠
+𝑒
+conditional
+superscript
+subscript
+𝑧
+𝑖
+1
+1
+⋯
+𝑘
+𝑠
+[z^{(1)},\cdots,z^{(k)}]\sim p_{\theta}^{propose}(z_{i+1}^{(1\cdots k)}\mid s)
+. This works better when the thought space is more constrained (e.g. each thought is just a word or a line), so proposing different thoughts in the same context avoids duplication.
+3. State evaluator
+V
+​
+(
+p
+θ
+,
+S
+)
+𝑉
+subscript
+𝑝
+𝜃
+𝑆
+V(p_{\theta},S)
+.
+Given a frontier of different states, the state evaluator evaluates the progress they make towards solving the problem, serving as a
+heuristic
+for the search algorithm to determine which states to keep exploring and in which order. While heuristics are a standard approach to solving search problems, they are typically either programmed (e.g. DeepBlue
+[
+3
+]
+) or learned (e.g. AlphaGo
+[
+29
+]
+). We propose a third alternative, by using the LM to deliberately reason about states. When applicable, such a deliberate heuristic can be more flexible than programmed rules, and more sample-efficient than learned models.
+Similar to the thought generator, we consider two strategies to evaluate states either independently or together:
+(a)
+Value
+each state independently:
+V
+​
+(
+p
+θ
+,
+S
+)
+​
+(
+s
+)
+∼
+p
+θ
+v
+​
+a
+​
+l
+​
+u
+​
+e
+​
+(
+v
+|
+s
+)
+​
+∀
+s
+∈
+S
+similar-to
+𝑉
+subscript
+𝑝
+𝜃
+𝑆
+𝑠
+superscript
+subscript
+𝑝
+𝜃
+𝑣
+𝑎
+𝑙
+𝑢
+𝑒
+conditional
+𝑣
+𝑠
+for-all
+𝑠
+𝑆
+V(p_{\theta},S)(s)\sim p_{\theta}^{value}(v|s)\ \forall s\in S
+, where a value prompt reasons about the state
+s
+𝑠
+s
+to generate a scalar value
+v
+𝑣
+v
+(e.g. 1-10) or a classification (e.g. sure/likely/impossible) that could be heuristically turned into a value. The basis of such evaluative reasoning can vary across problems and thought steps. In this work, we explore evaluation via few
+lookahead
+simulations (e.g. quickly confirm that 5, 5, 14 can reach 24 via 5 + 5 + 14, or “hot_l” can mean “inn” via filling “e” in “_”) plus commonsense (e.g. 1 2 3 are too small to reach 24, or no word can start with “tzxc”). While the former might promote “good” states, the latter could help eliminate “bad” states. Such valuations do not need to be perfect, and only need to be approximately helpful for decision making.
+(b)
+Vote
+across states:
+V
+​
+(
+p
+θ
+,
+S
+)
+​
+(
+s
+)
+=
+𝟙
+​
+[
+s
+=
+s
+∗
+]
+𝑉
+subscript
+𝑝
+𝜃
+𝑆
+𝑠
+1
+delimited-[]
+𝑠
+superscript
+𝑠
+V(p_{\theta},S)(s)=\mathds{1}[s=s^{*}]
+, where a “good” state
+s
+∗
+∼
+p
+θ
+v
+​
+o
+​
+t
+​
+e
+​
+(
+s
+∗
+|
+S
+)
+similar-to
+superscript
+𝑠
+superscript
+subscript
+𝑝
+𝜃
+𝑣
+𝑜
+𝑡
+𝑒
+conditional
+superscript
+𝑠
+𝑆
+s^{*}\sim p_{\theta}^{vote}(s^{*}|S)
+is voted out based on deliberately comparing different states in
+S
+𝑆
+S
+in a vote prompt.
+When problem success is harder to directly value (e.g. passage coherency), it is natural to to instead compare different partial solutions and vote for the most promising one. This is similar in spirit to a “step-wise” self-consistency strategy, i.e. cast “which state to explore” as a multi-choice QA, and use LM samples to vote for it.
+For both strategies, we could prompt the LM multiple times to aggregate the value or vote results to trade time/resource/cost for more faithful/robust heuristics.
+Algorithm 1
+ToT-BFS(
+x
+,
+p
+θ
+,
+G
+,
+k
+,
+V
+,
+T
+,
+b
+𝑥
+subscript
+𝑝
+𝜃
+𝐺
+𝑘
+𝑉
+𝑇
+𝑏
+x,p_{\theta},G,k,V,T,b
+)
+Input
+x
+𝑥
+x
+, LM
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+, thought generator
+G
+​
+(
+)
+𝐺
+G()
+& size limit
+k
+𝑘
+k
+, states evaluator
+V
+​
+(
+)
+𝑉
+V()
+, step limit
+T
+𝑇
+T
+, breadth limit
+b
+𝑏
+b
+.
+S
+0
+←
+{
+x
+}
+←
+subscript
+𝑆
+0
+𝑥
+S_{0}\leftarrow\{x\}
+for
+t
+=
+1
+,
+⋯
+,
+T
+𝑡
+1
+⋯
+𝑇
+t=1,\cdots,T
+do
+S
+t
+′
+←
+{
+[
+s
+,
+z
+]
+∣
+s
+∈
+S
+t
+−
+1
+,
+z
+t
+∈
+G
+​
+(
+p
+θ
+,
+s
+,
+k
+)
+}
+←
+subscript
+superscript
+𝑆
+′
+𝑡
+conditional-set
+𝑠
+𝑧
+formulae-sequence
+𝑠
+subscript
+𝑆
+𝑡
+1
+subscript
+𝑧
+𝑡
+G
+subscript
+𝑝
+𝜃
+𝑠
+𝑘
+S^{\prime}_{t}\leftarrow\{[s,z]\mid s\in S_{t-1},z_{t}\in{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathrm{G}}(p_{\theta},s,k)\}
+V
+t
+←
+V
+​
+(
+p
+θ
+,
+S
+t
+′
+)
+←
+subscript
+𝑉
+𝑡
+𝑉
+subscript
+𝑝
+𝜃
+subscript
+superscript
+𝑆
+′
+𝑡
+V_{t}\leftarrow V(p_{\theta},S^{\prime}_{t})
+S
+t
+←
+arg
+⁡
+max
+S
+⊂
+S
+t
+′
+,
+|
+S
+|
+=
+b
+​
+∑
+s
+∈
+S
+V
+t
+​
+(
+s
+)
+←
+subscript
+𝑆
+𝑡
+subscript
+formulae-sequence
+𝑆
+subscript
+superscript
+𝑆
+′
+𝑡
+𝑆
+𝑏
+subscript
+𝑠
+𝑆
+subscript
+𝑉
+𝑡
+𝑠
+S_{t}\leftarrow\arg\max_{S\subset S^{\prime}_{t},|S|=b}\sum_{s\in S}V_{t}(s)
+end
+for
+return
+G
+​
+(
+p
+θ
+,
+arg
+⁡
+max
+s
+∈
+S
+T
+⁡
+V
+T
+​
+(
+s
+)
+,
+1
+)
+𝐺
+subscript
+𝑝
+𝜃
+subscript
+𝑠
+subscript
+𝑆
+𝑇
+subscript
+𝑉
+𝑇
+𝑠
+1
+G(p_{\theta},\arg\max_{s\in S_{T}}V_{T}(s),1)
+Algorithm 2
+ToT-DFS(
+s
+,
+t
+,
+p
+θ
+,
+G
+,
+k
+,
+V
+,
+T
+,
+v
+t
+​
+h
+𝑠
+𝑡
+subscript
+𝑝
+𝜃
+𝐺
+𝑘
+𝑉
+𝑇
+subscript
+𝑣
+𝑡
+ℎ
+s,t,p_{\theta},G,k,V,T,v_{\small th}
+)
+Current state
+s
+𝑠
+s
+, step
+t
+𝑡
+t
+, LM
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+, thought generator
+G
+​
+(
+)
+𝐺
+G()
+and size limit
+k
+𝑘
+k
+, states evaluator
+V
+​
+(
+)
+𝑉
+V()
+, step limit
+T
+𝑇
+T
+, threshold
+v
+t
+​
+h
+subscript
+𝑣
+𝑡
+ℎ
+v_{\small th}
+if
+t
+>
+T
+𝑡
+𝑇
+t>T
+then
+record output
+G
+​
+(
+p
+θ
+,
+s
+,
+1
+)
+𝐺
+subscript
+𝑝
+𝜃
+𝑠
+1
+G(p_{\theta},s,1)
+end
+if
+for
+s
+′
+∈
+G
+​
+(
+p
+θ
+,
+s
+,
+k
+)
+superscript
+𝑠
+′
+𝐺
+subscript
+𝑝
+𝜃
+𝑠
+𝑘
+s^{\prime}\in G(p_{\theta},s,k)
+do
+▷
+▷
+\triangleright
+sorted candidates
+if
+V
+​
+(
+p
+θ
+,
+{
+s
+′
+}
+)
+​
+(
+s
+)
+>
+v
+t
+​
+h
+​
+r
+​
+e
+​
+s
+𝑉
+subscript
+𝑝
+𝜃
+superscript
+𝑠
+′
+𝑠
+subscript
+𝑣
+𝑡
+ℎ
+𝑟
+𝑒
+𝑠
+V(p_{\theta},\{s^{\prime}\})(s)>v_{\small thres}
+then
+▷
+▷
+\triangleright
+pruning
+DFS
+(
+s
+′
+,
+t
++
+1
+)
+superscript
+𝑠
+′
+𝑡
+1
+(s^{\prime},t+1)
+end
+if
+end
+for
+4. Search algorithm.
+Finally, within the ToT framework, one can plug and play different search algorithms depending on the tree structure. We explore two relatively simple search algorithms and leave more advanced ones (e.g. A*
+[
+11
+]
+, MCTS
+[
+2
+]
+) for future work:
+(a)
+Breadth-first search (BFS)
+(Algorithm
+1
+) maintains a set of the
+b
+𝑏
+b
+most promising states per step. This is used for Game of 24 and Creative Writing where the tree depth is limit (
+T
+≤
+3
+𝑇
+3
+T\leq 3
+), and initial thought steps can be evaluated and pruned to a small set (
+b
+≤
+5
+𝑏
+5
+b\leq 5
+).
+(b)
+Depth-first search (DFS)
+(Algorithm
+2
+) explores the most promising state first, until the final output is reached (
+t
+>
+T
+𝑡
+𝑇
+t>T
+), or the state evaluator deems it impossible to solve the problem from the current
+s
+𝑠
+s
+(
+V
+​
+(
+p
+θ
+,
+{
+s
+}
+)
+​
+(
+s
+)
+≤
+v
+t
+​
+h
+𝑉
+subscript
+𝑝
+𝜃
+𝑠
+𝑠
+subscript
+𝑣
+𝑡
+ℎ
+V(p_{\theta},\{s\})(s)\leq v_{th}
+for a value threshold
+v
+t
+​
+h
+subscript
+𝑣
+𝑡
+ℎ
+v_{th}
+). In the latter case, the subtree from
+s
+𝑠
+s
+is
+pruned
+to trade exploration for exploitation. In both cases, DFS
+backtracks
+to the parent state of
+s
+𝑠
+s
+to continue exploration.
+Conceptually, ToT has several benefits as a method for general problem-solving with LMs: (1)
+Generality.
+IO, CoT, CoT-SC, and self-refinement can be seen as special cases of ToT (i.e. trees of limited depth and breadth; Figure
+1
+). (2)
+Modularity.
+The base LM, as well as the thought decomposition, generation, evaluation, and search procedures can all be varied independently. (3)
+Adaptability
+. Different problem properties, LM capabilities, and resource constraints can be accommodated. (4)
+Convenience.
+No extra training is needed, just a pre-trained LM is sufficient. The next section will show how these conceptual benefits translate to strong empirical performance in different problems.
+4
+Experiments
+Game of 24
+Creative Writing
+5x5 Crosswords
+Input
+4 numbers
+(4 9 10 13)
+4 random sentences
+10 clues
+(h1. presented;..)
+Output
+An equation to reach 24
+(13-9)*(10-4)=24
+A passage of 4 paragraphs ending in the 4 sentences
+5x5 letters:
+SHOWN; WIRRA; AVAIL; …
+Thoughts
+3 intermediate equations
+(13-9=4 (left 4,4,10); 10-4=6 (left 4,6); 4*6=24)
+A short writing plan
+(1. Introduce a book that connects…)
+Words to fill in for clues:
+(h1. shown; v5. naled; …)
+#ToT steps
+3
+1
+5-10 (variable)
+Table 1:
+Task overview. Input, output, thought examples are in blue.
+We propose three tasks that are hard even when sampling from the state-of-the-art language model, GPT-4
+[
+23
+]
+, using standard IO prompting or chain-of-thought (CoT) prompting. We show how deliberate search in trees of thoughts (ToT) produces better results, and more importantly, interesting and promising new ways to use language models to solve problems requiring search or planning.
+Unless otherwise stated, we perform experiments using a Chat Completion mode GPT-4
+1
+1
+1
+Experiments were done between May 5-16, 2023.
+with a sampling temperature of 0.7.
+4.1
+Game of 24
+Game of 24 is a mathematical reasoning challenge, where the goal is to use 4 numbers and basic arithmetic operations (+-*/) to obtain 24.
+For example, given input “4 9 10 13”, a solution output could be “(10 - 4) * (13 - 9) = 24”.
+Figure 2:
+ToT in a game of 24. The LM is prompted for (a) thought generation and (b) valuation.
+Method
+Success
+IO prompt
+7.3%
+CoT prompt
+4.0%
+CoT-SC
+(k=100)
+9.0%
+ToT (ours)
+(b=1)
+45%
+ToT (ours)
+(b=5)
+74%
+IO + Refine
+(k=10)
+27%
+IO
+(best of 100)
+33%
+CoT
+(best of 100)
+49%
+Table 2:
+Game of 24 Results.
+Figure 3:
+Game of 24 (a) scale analysis & (b) error analysis.
+Task Setup.
+We scrape data from
+4nums.com
+, which has 1,362 games that are sorted from easy to hard by human solving time, and use a subset of relatively hard games indexed 901-1,000 for testing. For each task, we consider the output as success if it is a valid equation that equals 24 and uses the input numbers each exactly once. We report the success rate across 100 games as the metric.
+Baselines.
+We use a standard input-output (IO) prompt with 5 in-context examples. For chain-of-thought (CoT) prompting, we augment each input-output pair with 3 intermediate equations, each operating on two remaining numbers. For example, given input “4 9 10 13”, the thoughts could be “13 - 9 = 4 (left: 4 4 10); 10 - 4 = 6 (left: 4 6); 4 * 6 = 24 (left: 24)”. For each game, we sample IO and CoT prompting for 100 times for average performance.
+We also consider a CoT self-consistency baseline, which takes the majority output from 100 CoT samples, and an iterative-refine approach on top of an IO sample for at most
+10
+10
+10
+iterations. At each iteration, the LM is conditioned on all previous history to “reflect on your mistakes and generate a refined answer” if the output is incorrect. Note that it uses groundtruth feedback signals about equation correctness.
+ToT Setup.
+To frame Game of 24 into ToT, it is natural to decompose the thoughts into 3 steps, each an intermediate equation. As shown in Figure
+2
+(a), at each tree node, we exact the remaining numbers and prompt the LM to propose some possible next steps.
+The same “propose prompt” is used for all 3 thought steps, though it only has one example with 4 input numbers.
+We perform a breadth-first search (BFS) in ToT, where at each step we keep the best
+b
+=
+5
+𝑏
+5
+b=5
+candidates.
+To perform deliberate BFS in ToT, as shown in Figure
+2
+(b), we prompt LM to evaluate each thought candidate as “sure/maybe/impossible” with regard to reaching 24. The aim is to promote correct partial solutions that can be verdicted within few lookahead trials, and eliminate impossible partial solutions based on “too big/small” commonsense, and keep the rest “maybe”. We sample values
+3
+3
+3
+times for each thought.
+Results.
+As shown in Table
+3
+, IO, CoT, and CoT-SC prompting methods perform badly on the task, achieving only 7.3%, 4.0%, and 9.0% success rates. In contrast, ToT with a breadth of
+b
+=
+1
+𝑏
+1
+b=1
+already achieves a success rate of
+45
+%
+percent
+45
+45\%
+, while
+b
+=
+5
+𝑏
+5
+b=5
+achieves
+74
+%
+percent
+74
+74\%
+.
+We also consider an oracle setup for IO/CoT, by calculating the success rate using best of
+k
+𝑘
+k
+samples
+(
+1
+≤
+k
+≤
+100
+)
+1
+𝑘
+100
+(1\leq k\leq 100)
+. To compare IO/CoT (best of k) with ToT, we consider calculating the tree nodes visited per task in ToT across
+b
+=
+1
+​
+⋯
+​
+5
+𝑏
+1
+⋯
+5
+b=1\cdots 5
+, and map the 5 success rates in Figure
+3
+(a), treating IO/CoT (best of
+k
+𝑘
+k
+) as visiting
+k
+𝑘
+k
+nodes in a bandit. Not surprisingly, CoT scales better than IO, and best of 100 CoT samples achieve a success rate of
+49
+%
+percent
+49
+49\%
+, but still much worse than exploring more nodes in ToT (
+b
+>
+1
+𝑏
+1
+b>1
+).
+Error analysis.
+Figure
+3
+(b) breaks down at which step CoT and ToT samples fail the task, i.e. the thought (in CoT) or all
+b
+𝑏
+b
+thoughts (in ToT) are invalid or impossible to reach 24. Notably, around 60% of CoT samples already failed the task after generating the first step, or equivalently, the first three words (e.g. “
+4
++
+9
+4
+9
+4+9
+”). This highlights the issues with direct left-to-right decoding.
+4.2
+Creative writing
+Next, we invent a creative writing task where the input is 4 random sentences and the output should be a coherent passage with 4 paragraphs that end in the 4 input sentences respectively.
+Such a task is open-ended and exploratory, and challenges creative thinking as well as high-level planning.
+Task setup.
+We sample random sentences from
+randomwordgenerator.com
+to form 100 inputs, and there is no groundtruth passage for each input constraint. As we find that GPT-4 can follow the input constraints most of the time, we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods. For the former, we sample 5 scores and average them for each task output, and we find these 5 scores usually consistent, with a standard deviation of around
+0.56
+0.56
+0.56
+on average across outputs. For the latter, we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.
+Baselines.
+Given the creative nature of the task, both IO and CoT prompts are zero-shot. While the former prompts the LM to directly generate a coherent passage given input constraints, the latter prompts the LM to first make a brief plan then write the passage, i.e. the plan serves as the intermediate thought step. We generate 10 IO and CoT samples per task.
+We also consider an iterative-refine (
+k
+≤
+5
+𝑘
+5
+k\leq 5
+) method on top of a random IO sample for each task, where the LM is conditioned on input constraints and the last generated passage to decide if the passage is already “perfectly coherent”, and if not generate a refined one.
+ToT setup.
+We build a ToT with depth 2 (and only 1 intermediate thought step) — the LM first generates
+k
+=
+5
+𝑘
+5
+k=5
+plans and votes for the best one (Figure
+4
+), then similarly generate
+k
+=
+5
+𝑘
+5
+k=5
+passages based on the best plan then vote for the best one. Here the breadth limit
+b
+=
+1
+𝑏
+1
+b=1
+, as only one choice is kept per step. A simple zero-shot vote prompt (“analyze choices below, then conclude which is most promising for the instruction”) is used to sample 5 votes at both steps.
+Results.
+Figure
+5
+(a) shows average GPT-4 scores across 100 tasks, where ToT (7.56) is deemed to generate more coherent passages than IO (6.19) and CoT (6.93) on average. While such an automatic metric might be noisy, Figure
+5
+(b) confirms the finding by showing that humans prefer ToT over CoT in 41 out of 100 passage pairs, while only prefer CoT over ToT in 21 (other 38 pairs are found “similarly coherent”). Lastly, iterative-refine is more effective on this natural language task, where it improves IO coherency score from 6.19 to 7.67, and ToT coherency score from 7.56 to 7.91.
+We believe it could be thought of as a third approach to thought generation in the ToT framework, where new thoughts can arise from refining old thoughts instead of i.i.d. or sequentially generated.
+Figure 4:
+A step of deliberate search in a randomly picked Creative Writing task. Given the input, the LM samples 5 different plans, then votes 5 times to decide which plan is best. The majority choice is used to consequently write the output passage with the same sample-vote procedure.
+Figure 5:
+Creative Writing results.
+Method
+Success Rate (%)
+Letter
+Word
+Game
+IO
+38.7
+14
+0
+CoT
+40.6
+15.6
+1
+ToT (ours)
+78
+60
+20
++best state
+82.4
+67.5
+35
+-prune
+65.4
+41.5
+5
+-backtrack
+54.6
+20
+5
+Table 3:
+Mini Crosswords results.
+4.3
+Mini crosswords
+Figure 6:
+In Mini Crosswords, (a) how thoughts are proposed and aggregated in a priority queue for depth-first search (DFS), and (b) how a state is evaluated based on the possibility of filling in each remaining word clue, and pruned if any remaining clue is deemed not possible to fill by the LM. Then DFS backtracks to the parent state and explore the next promising thought for clue.
+In Game of 24 and Creative Writing, ToT is relatively shallow — at most 3 thought steps are needed to reach the final output. Here we explore
+5
+×
+5
+5
+5
+5\times 5
+mini crosswords as a harder search problem involving natural language. Again, the goal is not just to solve the task, as more general crosswords can be readily solved with specialized NLP pipelines
+[
+34
+]
+that leverages large-scale retrieval instead of LM. Rather, we aim to explore the limit of LM as a general problem solver that explores its own thoughts and guides its own exploration with deliberate reasoning as heuristics.
+Task setup.
+We scrape data from
+GooBix
+, which contains 156 games of
+5
+×
+5
+5
+5
+5\times 5
+mini crosswords. As we observe adjacent games contain similar clues, we use 20 games with indices
+1
+,
+6
+,
+⋯
+,
+91
+,
+96
+1
+6
+⋯
+91
+96
+1,6,\cdots,91,96
+for testing, and games
+136
+,
+141
+,
+146
+,
+151
+,
+156
+136
+141
+146
+151
+156
+136,141,146,151,156
+for prompting.
+For each task, the input describes the 5 horizontal clues and 5 vertical clues, and the output should be a board of
+5
+×
+5
+=
+25
+5
+5
+25
+5\times 5=25
+letters to solve the crosswords. For evaluation, we consider three levels of success: the portion of correct letters (25 per game), words (10 per game), and games.
+Baselines.
+We provide 5 example input-output pairs in the IO prompt, and in the CoT prompt additionally include intermediate words in the order h1..5 then v1..5. We run each prompt for 10 samples and average the results.
+ToT setup.
+We leverage a depth-first search (Algorithm
+2
+) that keeps exploring the most promising subsequent word clue until the state is no longer promising, then backtrack to the parent state to explore alternative thoughts.
+To make search tractable, subsequent thoughts are constrained not to change any filled words or letters, so that the ToT has at most 10 intermediate steps.
+For thought generation, at each state we translate all existing thoughts (e.g. “h2.motor; h1.tasks” for the state in Figure
+6
+(a)) into letter constraints for remaining clues (e.g. “v1.To heap: tm___;…”) and prompt a proposal prompt
+5
+5
+5
+times to come up with candidates for where and what to fill in the next word. Importantly, we also prompt the LM to give a confidence level for different thoughts, and aggregate these across proposals to obtain a sorted list of next thoughts to explore (Figure
+6
+(a)).
+For state evaluations, we similarly translate each state into letter constraints for remaining clues, then evaluate for each clue if it is possible to fill given the constraints. If any remaining clue is deemed “impossible” to fill in (e.g. “v1. To heap: tm_s_”), then the exploration of the state’s subtree is pruned and DFS backtracks to its parent to explore the next promising thought. We limit DFS search steps to 100, and simply render the deepest explored state (the first explored one if multiple) into the final output.
+Results.
+As shown in Table
+5
+, IO and CoT prompting methods perform poorly with a word-level success rate less than
+16
+%
+percent
+16
+16\%
+, while ToT significantly improves all metrics, achieving a word-level success rate of
+60
+%
+percent
+60
+60\%
+and solving 4 out of 20 games. Such an improvement is not surprising, given IO and CoT lack mechanisms to try different clues, make changes to decisions, or backtrack.
+Oracle and ablation studies.
+When outputting from the oracle best DFS state (instead of the heuristically determined best state) per task, ToT performance is even higher and actually solves 7/20 games (Table
+5
+, “+best state”), indicating our simple output heuristics can be readily improved. Interestingly, sometimes when the crosswords game is actually solved, the state evaluator might still deem some words as “impossible” and prune — possibly because
+5
+×
+5
+5
+5
+5\times 5
+crosswords by design have some rare or obselete words that GPT-4 cannot recognize
+2
+2
+2
+For example, “agend” is an obsolete form of “agendum”, but GPT-4 deems it a typo for “agenda”. External retrieval
+or web interaction
+could augment LM for problem solving under knowledge uncertainty.
+.
+Given the state evaluation as a pruning heuristic is imperfect, we also explore ablating the pruning, and find the performance generally worse (Table
+5
+, “-prune”). However, it could actually find the correct solution for 4/20 games (though only outputting 1 via heuristic), 3 of which are games ToT+pruning cannot solve within 100 steps. Thus, better heuristics for DFS pruning are critical for problem solving in this case.
+Lastly, we confirm the importance of backtracking by running an ablation that keeps filling the most promising clue for at most 20 steps, allowing overwrites. This is similar to a “greedy” BFS search with breadth limit of
+b
+=
+1
+𝑏
+1
+b=1
+, and performs poorly with a word level success of only
+20
+%
+percent
+20
+20\%
+(Table
+5
+, “-backtrack”).
+5
+Related Work
+Planning and decision making.
+Smart planning and decision making are critical to achieving predefined goals. As they are trained on vast amount of world knowledge and human examples,
+LMs are known to have already absorbed rich commonsense that makes it possible to propose reasonable plans conditioned on problem setting and environmental states
+[
+12
+,
+42
+,
+37
+,
+13
+,
+35
+,
+41
+,
+40
+]
+. Our proposed ToT approach extends existing planning formulations by considering multiple potentially feasible plans simultaneously at each problem-solving step, and proceeding with the most promising ones. The integration between thought sampling and value feedback organically integrates planning and decision-making mechanisms, enabling effective search inside a solution tree. On the other hand, traditional decision-making procedures usually require training dedicated reward and policy models as in reinforcement learning (for example CHAI
+[
+33
+]
+), whereas we use the LM itself to provide the value estimates for decision making.
+RAP
+[
+9
+]
+is a concurrent work that treats language model reasoning as planning with its internal world model, and proposes a MCTS-based method similar to ToT. However, its tasks are simpler than ours, and its framework lacks the modularity to incorporate different tree search algorithms.
+Self-reflection.
+Using LLMs to assess the viability of their own predictions is becoming an increasingly important procedure in problem solving.
+[
+28
+,
+20
+,
+24
+]
+introduced the “self-reflection” mechanism, in which LMs provide feedback to their generation candidates.
+[
+4
+]
+improves LMs code generation accuracy by injecting feedback messages generated by the LM itself based on its code execution results. Similarly,
+[
+17
+]
+also introduces “critic” or review steps over the actions and states, deciding the next action to take in solving computer operation tasks. Another recent work very relevant to ours is “self-eval guided decoding”
+[
+39
+]
+. Similar to our method, self-eval decoding also follows a tree-search procedure with leaves sampled from stochastic beam search decoding, which are then evaluated by LLM itself with carefully prepared self-eval prompts. Their approach however, uses the PAL formulation
+[
+8
+]
+which represents thoughts as codes, which makes it difficult to tackle challenging tasks like creative writing which we consider in this paper. Our Tree-of-Thought formulation is thus more versatile and handles challenging tasks on which GPT-4 only achieves very low accuracy with standard prompts.
+Program-guided LLM generation.
+Our proposal is also related to recent advancements that organize LM’s behavior with systematic procedures
+[
+14
+,
+44
+,
+6
+,
+43
+]
+or symbolic program guidance. For example,
+Schlag et al. [
+27
+]
+embeds LMs in an algorithmic search procedure to help solve problems like question answering step-by-step, in which the search trees are expanded by relevant paragraphs that might provide answers. This approach however differs from ours in that trees are expanded by sampling external paragraphs instead of the LM’s own thoughts, and there is no reflection or voting steps. Another approach, LLM+P
+[
+18
+]
+, goes one step further and delegates the actual planning process to a classical planner.
+Classical search methods.
+Last but not least, our approach can be treated as a modern rendition of classical search methods for problem solving. For example it can be considered as a heuristic search algorithm like A*
+[
+10
+]
+, in which the heuristic at each search node is provided by the LM’s self-assessment. From this perspective, our method is also related to NeuroLogic A*esque decoding
+[
+19
+]
+, which is inspired by A* search but introduces look-ahead heuristics that are efficient for LMs to improve the beam-search or top-k sampling decoding. This method however is constrained to sentence generation tasks, whereas our framework are designed for complex, multi-step problem solving guarded by value feedback.
+6
+Discussion
+Limitations and future directions.
+Deliberate search such as ToT might not be necessary for many existing tasks that GPT-4 already excels at (see Appendix
+B.1
+), and as an initial step this work only explores three relatively simple tasks that challenges GPT-4 (see Appendix
+B.2
+for some GPT-3.5 experiment results) and calls of better search and planning abilities incorporated with LMs. However, as we begin to deploy LMs for more real-world decision making applications (e.g. coding, data analysis, robotics, etc.), more complex tasks could emerge and present new opportunities to study these research questions. Also, search methods like ToT requires more resources (e.g. GPT-4 API cost) than sampling methods in order to improve task performances, but the modular flexibility of ToT allows users to customize such performance-cost tradeoffs, and ongoing open-source efforts
+[
+32
+]
+should readily reduce such costs in the near future. More details about cost and efficiency are in Appendix
+B.3
+. Lastly, this work focuses on using an off-the-shelf LM, and fine-tuning LMs using a ToT-style high-level counterfactual decision making (e.g. deliberating over potential choices for the next paragraph, instead of predicting the next token) might present opportunities to enhance the problem-solving capabilities of LMs.
+Conclusion.
+The associative “System 1” of LMs can be beneficially augmented by a “System 2” based on searching a tree of possible paths to the solution to a problem. The Tree of Thoughts framework provides a way to translate classical insights about problem-solving into actionable methods for contemporary LMs. At the same time, LMs address a weakness of these classical methods, providing a way to solve complex problems that are not easily formalized, such as creative writing. We see this intersection of LMs with classical approaches to AI as an exciting direction.
+Broader Impact
+ToT is a framework that empowers LMs to more autonomously and intelligently make decisions and solve problems. While current tasks are limited to reasoning and search problems, future applications involving interaction with external environments or humans could bring potential danger, e.g. facilitating harmful uses of LMs. On the other hand, ToT also improves the interpretability of model decisions and the opportunity for human alignment, as the resulting representations are readable, high-level language reasoning instead of implicit, low-level token values.
+Acknowledgements
+SY and KN acknowledge support from an Oracle Collaborative Research award and the National Science Foundation under Grant No. 2239363. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. SY is also supported by the Harold W. Dodds Fellowship from Princeton.
+References
+Brown et al. [2020]
+T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal,
+A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.
+Language models are few-shot learners.
+Advances in neural information processing systems
+,
+33:1877–1901, 2020.
+Browne et al. [2012]
+C. Browne, E. J. Powley, D. Whitehouse, S. M. M. Lucas, P. I. Cowling,
+P. Rohlfshagen, S. Tavener, D. P. Liebana, S. Samothrakis, and S. Colton.
+A survey of monte carlo tree search methods.
+IEEE Transactions on Computational Intelligence and AI in
+Games
+, 4:1–43, 2012.
+Campbell et al. [2002]
+M. Campbell, A. J. Hoane Jr, and F.-h. Hsu.
+Deep blue.
+Artificial intelligence
+, 134(1-2):57–83,
+2002.
+Chen et al. [2023]
+X. Chen, M. Lin, N. Schärli, and D. Zhou.
+Teaching large language models to self-debug, 2023.
+Chowdhery et al. [2022]
+A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham,
+H. W. Chung, C. Sutton, S. Gehrmann, et al.
+Palm: Scaling language modeling with pathways.
+arXiv preprint arXiv:2204.02311
+, 2022.
+Creswell and Shanahan [2022]
+A. Creswell and M. Shanahan.
+Faithful reasoning using large language models.
+arXiv preprint arXiv:2208.14271
+, 2022.
+Daw et al. [2005]
+N. D. Daw, Y. Niv, and P. Dayan.
+Uncertainty-based competition between prefrontal and dorsolateral
+striatal systems for behavioral control.
+Nature neuroscience
+, 8(12):1704–1711,
+2005.
+Gao et al. [2023]
+L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig.
+Pal: Program-aided language models, 2023.
+Hao et al. [2023]
+S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu.
+Reasoning with language model is planning with world model.
+arXiv preprint arXiv:2305.14992
+, 2023.
+Hart et al. [1968a]
+P. E. Hart, N. J. Nilsson, and B. Raphael.
+A formal basis for the heuristic determination of minimum cost paths.
+IEEE Transactions on Systems Science and Cybernetics
+,
+4(2):100–107, 1968a.
+doi:
+10.1109/TSSC.1968.300136
+.
+Hart et al. [1968b]
+P. E. Hart, N. J. Nilsson, and B. Raphael.
+A formal basis for the heuristic determination of minimum cost paths.
+IEEE transactions on Systems Science and Cybernetics
+,
+4(2):100–107, 1968b.
+Huang et al. [2022a]
+W. Huang, P. Abbeel, D. Pathak, and I. Mordatch.
+Language models as zero-shot planners: Extracting actionable
+knowledge for embodied agents, 2022a.
+Huang et al. [2022b]
+W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson,
+I. Mordatch, Y. Chebotar, et al.
+Inner monologue: Embodied reasoning through planning with language
+models.
+arXiv preprint arXiv:2207.05608
+, 2022b.
+Jung et al. [2022]
+J. Jung, L. Qin, S. Welleck, F. Brahman, C. Bhagavatula, R. L. Bras, and
+Y. Choi.
+Maieutic prompting: Logically consistent reasoning with recursive
+explanations.
+arXiv preprint arXiv:2205.11822
+, 2022.
+Kahneman [2011]
+D. Kahneman.
+Thinking, fast and slow
+.
+Macmillan, 2011.
+Kahneman et al. [2002]
+D. Kahneman, S. Frederick, et al.
+Representativeness revisited: Attribute substitution in intuitive
+judgment.
+Heuristics and biases: The psychology of intuitive judgment
+,
+49(49-81):74, 2002.
+Kim et al. [2023]
+G. Kim, P. Baldi, and S. McAleer.
+Language models can solve computer tasks, 2023.
+Liu et al. [2023]
+B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone.
+Llm+p: Empowering large language models with optimal planning
+proficiency, 2023.
+Lu et al. [2021]
+X. Lu, S. Welleck, P. West, L. Jiang, J. Kasai, D. Khashabi, R. L. Bras,
+L. Qin, Y. Yu, R. Zellers, N. A. Smith, and Y. Choi.
+Neurologic a*esque decoding: Constrained text generation with
+lookahead heuristics.
+In
+North American Chapter of the Association for Computational
+Linguistics
+, 2021.
+Madaan et al. [2023]
+A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon,
+N. Dziri, S. Prabhumoye, Y. Yang, S. Welleck, B. P. Majumder, S. Gupta,
+A. Yazdanbakhsh, and P. Clark.
+Self-refine: Iterative refinement with self-feedback, 2023.
+Newell et al. [1959]
+A. Newell, J. C. Shaw, and H. A. Simon.
+Report on a general problem solving program.
+In
+IFIP congress
+, volume 256, page 64. Pittsburgh, PA, 1959.
+Newell et al. [1972]
+A. Newell, H. A. Simon, et al.
+Human problem solving
+.
+Prentice-Hall, 1972.
+OpenAI [2023]
+OpenAI.
+Gpt-4 technical report.
+ArXiv
+, abs/2303.08774, 2023.
+Paul et al. [2023]
+D. Paul, M. Ismayilzada, M. Peyrard, B. Borges, A. Bosselut, R. West, and
+B. Faltings.
+Refiner: Reasoning feedback on intermediate representations, 2023.
+Radford et al. [2018]
+A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al.
+Improving language understanding by generative pre-training.
+OpenAI blog
+, 2018.
+Radford et al. [2019]
+A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.
+Language models are unsupervised multitask learners.
+OpenAI blog
+, 1(8):9, 2019.
+Schlag et al. [2023]
+I. Schlag, S. Sukhbaatar, A. Celikyilmaz, W. tau Yih, J. Weston,
+J. Schmidhuber, and X. Li.
+Large language model programs, 2023.
+Shinn et al. [2023]
+N. Shinn, B. Labash, and A. Gopinath.
+Reflexion: an autonomous agent with dynamic memory and
+self-reflection, 2023.
+Silver et al. [2017]
+D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez,
+T. Hubert, L. Baker, M. Lai, A. Bolton, et al.
+Mastering the game of go without human knowledge.
+nature
+, 550(7676):354–359, 2017.
+Sloman [1996]
+S. A. Sloman.
+The empirical case for two systems of reasoning.
+Psychological bulletin
+, 119(1):3, 1996.
+Stanovich [1999]
+K. E. Stanovich.
+Who is rational? Studies of individual differences in
+reasoning
+.
+Psychology Press, 1999.
+Touvron et al. [2023]
+H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix,
+B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.
+Llama: Open and efficient foundation language models.
+arXiv preprint arXiv:2302.13971
+, 2023.
+Verma et al. [2022]
+S. Verma, J. Fu, S. Yang, and S. Levine.
+Chai: A chatbot ai for task-oriented dialogue with offline
+reinforcement learning.
+In
+Proceedings of the 2022 Conference of the North American
+Chapter of the Association for Computational Linguistics: Human Language
+Technologies
+, pages 4471–4491, 2022.
+Wallace et al. [2022]
+E. Wallace, N. Tomlin, A. Xu, K. Yang, E. Pathak, M. Ginsberg, and D. Klein.
+Automated crossword solving.
+arXiv preprint arXiv:2205.09665
+, 2022.
+Wang et al. [2023a]
+L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim.
+Plan-and-solve prompting: Improving zero-shot chain-of-thought
+reasoning by large language models, 2023a.
+Wang et al. [2022]
+X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou.
+Self-consistency improves chain of thought reasoning in language
+models.
+arXiv preprint arXiv:2203.11171
+, 2022.
+Wang et al. [2023b]
+Z. Wang, S. Cai, A. Liu, X. Ma, and Y. Liang.
+Describe, explain, plan and select: Interactive planning with large
+language models enables open-world multi-task agents, 2023b.
+Wei et al. [2022]
+J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou.
+Chain of thought prompting elicits reasoning in large language
+models.
+arXiv preprint arXiv:2201.11903
+, 2022.
+Xie et al. [2023]
+Y. Xie, K. Kawaguchi, Y. Zhao, X. Zhao, M.-Y. Kan, J. He, and Q. Xie.
+Decomposition enhances reasoning via self-evaluation guided decoding,
+2023.
+Yang et al. [2023]
+S. Yang, O. Nachum, Y. Du, J. Wei, P. Abbeel, and D. Schuurmans.
+Foundation models for decision making: Problems, methods, and
+opportunities, 2023.
+Yao et al. [2022]
+S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao.
+ReAct: Synergizing reasoning and acting in language models.
+arXiv preprint arXiv:2210.03629
+, 2022.
+Zhang et al. [2023]
+S. Zhang, Z. Chen, Y. Shen, M. Ding, J. B. Tenenbaum, and C. Gan.
+Planning with large language models for code generation.
+In
+The Eleventh International Conference on Learning
+Representations
+, 2023.
+URL
+https://openreview.net/forum?id=Lr8cOOtYbfL
+.
+Zhou et al. [2022]
+D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans,
+C. Cui, O. Bousquet, Q. Le, et al.
+Least-to-most prompting enables complex reasoning in large language
+models.
+arXiv preprint arXiv:2205.10625
+, 2022.
+Zhu et al. [2022]
+X. Zhu, J. Wang, L. Zhang, Y. Zhang, R. Gan, J. Zhang, and Y. Yang.
+Solving math word problem via cooperative reasoning induced language
+models.
+arXiv preprint arXiv:2210.16257
+, 2022.
+Appendix A
+Code, Prompts, Trajectories
+All code is available at
+https://github.com/princeton-nlp/tree-of-thought-llm
+.
+All prompts are available at
+https://github.com/princeton-nlp/tree-of-thought-llm/tree/master/src/tot/prompts
+.
+Trajectories are available at
+https://github.com/princeton-nlp/tree-of-thought-llm/tree/master/logs
+.
+Appendix B
+Additional Experiment Results
+Given the motivation of exploring and extending the capability frontier of language models, our experiments in the main paper have focused on a setup with the state-of-the-art language model (GPT-4), and three hard tasks invented to challenge it. Here, we report additional experiments with weaker LLM or easier tasks, and discuss cost and efficiency.
+GSM8K
+StrategyQA
+IO
+51
+73
+CoT
+86
+82
+ToT
+90
+83
+Table 4:
+New tasks with
+zero-shot ToT and GPT-4.
+GPT-4
+GPT-3.5
+IO
+7.3%
+6%
+CoT
+4.0%
+3%
+ToT
+74%
+19%
+Table 5:
+Game of 24 with
+GPT-4 vs GPT-3.5.
+GPT-4
+GPT-3.5
+IO
+6.19
+4.47
+CoT
+6.93
+5.16
+ToT
+7.56
+6.62
+Table 6:
+Creative Writing with
+GPT-4 vs. GPT-3.5.
+B.1
+Extension to new tasks (GSM8k, StrategyQA) with zero-shot ToT
+While more common NLP tasks might be too easy for GPT-4 and do not require ToT (which is why we considered harder new tasks), we believe applying ToT to new tasks could be straightforward. For example, we implemented a simple and generic zero-shot ToT-BFS similar to creative writing (sample 5 problem solving strategies then vote for the best one; then sample 5 solutions based on the best strategy then vote for the best one) for GSM8K and StrategyQA with few extra lines of code:
+# define the answer format of new tasks
+gsm8k_format = ‘"the answer is n" where n is a number’
+strategyqa_format = ‘either "the answer is yes" or "the answer is no"’
+
+# define zero-shot io prompting
+standard_prompt = ‘Answer the following question with {format}: {input}’
+
+# define thought format for zero-shot cot and zero-shot tot
+cot_prompt = ‘‘‘Answer the following question: {input}
+
+Make a strategy then write. Your output should be of the following format:
+
+Strategy:
+Your strategy about how to answer the question.
+
+Answer:
+Your answer to the question. It should end with {format}.
+’’’
+
+# define zero-shot voting used for zero-shot tot
+vote_prompt = ‘‘‘Given an instruction and several choices,
+decide which choice is most promising.
+Analyze each choice in detail, then conclude in the last line
+"The best choice is {s}", where s the integer id of the choice.
+’’’
+We evaluated on a subset of 100 random GSM8K test and StrategyQA dev questions. As shown in Table
+B
+and as expected, ToT improves over CoT on both tasks (but only slightly, given GPT-4 + CoT is already very good on such tasks, and StrategyQA’s bottleneck is external knowledge, not reasoning). Considering computational costs, it is more suitable to try smaller LLMs + ToT for traditional NLP tasks, or GPT-4 + ToT for hard tasks that challenge GPT-4 + CoT’s reasoning.
+B.2
+Extension to new LMs (GPT-3.5)
+To understand how ToT works with other LLMs, we also ran GPT-3.5-turbo for Creative Writing (Table
+B
+) and Game of 24 (Table
+B
+).
+On both tasks, “ToT
+>
+>
+CoT
+>
+>
+IO” remains true for GPT-3.5.
+On Creative Writing, we find GPT-3.5+ToT outperform GPT-4+IO, and similar to GPT-4+CoT, which suggests ToT could also work well on weaker language models.
+On Game of 24 (we changed 1-shot proposal prompt to 3-shot to make it work), GPT-3.5+ToT’s 19% is far worse than GPT-4+ToT’s 74%. To further understand the importance of generation vs. evaluation, we ran GPT-4 generation + GPT-3.5 evaluation (64%) and GPT-3.5 generation + GPT-4 evaluation (31%). This suggests the game’s bottleneck is thought generation, and different generation/evaluation language models might attain decent results while reducing costs.
+B.3
+Cost and efficiency
+Running ToT requires significantly more computations than IO or CoT prompting. For example, in Game of 24 (Table
+7
+below), solving a problem with ToT requires 5.5k completion tokens, close to 100 CoT trials (6.7k tokens). But the performance of ToT is better than best of 100 independent CoT trials.
+Game of 24
+Generate/Prompt tokens
+Cost per case
+Success
+IO (best of 100)
+1.8k / 1.0k
+$0.13
+33%
+CoT (best of 100)
+6.7k / 2.2k
+$0.47
+49%
+ToT
+5.5k / 1.4k
+$0.74
+74%
+Table 7:
+Cost analysis on Game of 24.
+On Creative Writing (Table
+8
+below), we found ToT takes around 5x completion tokens and money cost, which is intuitive as
+b
+=
+5
+𝑏
+5
+b=5
+and most tokens are generated passages.
+Creative Writing
+Generate/Prompt tokens
+Cost per case
+IO
+0.9k / 0.4k
+$0.06
+CoT
+0.9k / 0.4k
+$0.07
+ToT
+4k / 2.9k
+$0.32
+Table 8:
+Cost analysis on Game of 24.
+So completing Game of 24 and Creative Writing’s main ToT experiments cost around
+0.74
+×
+100
++
+0.32
+×
+100
+=
+106
+0.74
+100
+0.32
+100
+106
+0.74\times 100+0.32\times 100=106
+dollars. Crosswords’ DFS experiments should be also within
+100
+100
+100
+dollars. In general, cost and efficiency of ToT highly depend on the prompts and search algorithms used, and could require 5-100 times more generated tokens than CoT. Some actionable insights:
+•
+We recommend using ToT on tasks requiring deliberate reasoning, on which CoT struggles.
+•
+Flexibility of ToT allows some performance-cost tradeoff, e.g., change beam size or vote number in BFS, few-shot vs. zero-shot prompting, GPT-3.5 vs. GPT-4, etc. One could configure the setup based on some resource constraints or performance goal.
+•
+There is much space for improving efficiency, e.g., BFS could early stop when solution is found, or trim down beam size to when some thoughts are ”impossible”.
+•
+We believe that more computation is indeed required in order for the model to achieve stronger intelligence, and this should not become a blocking issue as in the long run, (open-source) LMs will become much cheaper and more efficient. It is also a great direction how to better train/finetune LMs for thought generation and/or evaluation.
+◄
+Feeling
+lucky?
+Conversion
+report
+Report
+an issue
+View original
+on arXiv
+►

research/notes/230510601-tree-of-thoughts-deliberate-problem-solving-with-large-language-models.md

@@ -0,0 +1,213 @@
+---
+title: '[2305.10601] Tree of Thoughts: Deliberate Problem Solving with Large Language
+  Models'
+id: 230510601-tree-of-thoughts-deliberate-problem-solving-with-large-language-models
+tags:
+- deepread
+created: '2026-06-10T00:39:55.627654Z'
+source: https://arxiv.org/abs/2305.10601
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:39:55.627506Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2305.10601] Tree of Thoughts: Deliberate Problem Solving with Large Language Models
+Computer Science > Computation and Language
+arXiv:2305.10601
+(cs)
+[Submitted on 17 May 2023 (
+v1
+), last revised 3 Dec 2023 (this version, v2)]
+Title:
+Tree of Thoughts: Deliberate Problem Solving with Large Language Models
+Authors:
+Shunyu Yao
+,
+Dian Yu
+,
+Jeffrey Zhao
+,
+Izhak Shafran
+,
+Thomas L. Griffiths
+,
+Yuan Cao
+,
+Karthik Narasimhan
+View a PDF of the paper titled Tree of Thoughts: Deliberate Problem Solving with Large Language Models, by Shunyu Yao and 6 other authors
+View PDF
+HTML (experimental)
+Abstract:
+Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role. To surmount these challenges, we introduce a new framework for language model inference, Tree of Thoughts (ToT), which generalizes over the popular Chain of Thought approach to prompting language models, and enables exploration over coherent units of text (thoughts) that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices. Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%. Code repo with all prompts:
+this https URL
+.
+Comments:
+NeurIPS 2023 camera ready version. Code repo with all prompts:
+this https URL
+Subjects:
+Computation and Language (cs.CL)
+; Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
+Cite as:
+arXiv:2305.10601
+[cs.CL]
+(or
+arXiv:2305.10601v2
+[cs.CL]
+for this version)
+https://doi.org/10.48550/arXiv.2305.10601
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Shunyu Yao [
+view email
+]
+[v1]
+Wed, 17 May 2023 23:16:17 UTC (609 KB)
+[v2]
+Sun, 3 Dec 2023 22:50:35 UTC (623 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Tree of Thoughts: Deliberate Problem Solving with Large Language Models, by Shunyu Yao and 6 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.CL
+< prev
+|
+next >
+new
+|
+recent
+|
+2023-05
+Change to browse by:
+cs
+cs.AI
+cs.LG
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+5 blog links
+(
+what is this?
+)
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+Links to Code Toggle
+Papers with Code
+(
+What is Papers with Code?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/231004406-language-agent-tree-search-unifies-reasoning-acting-and-planning-in-la-2.md

@@ -0,0 +1,4095 @@
+---
+title: '[2310.04406] Language Agent Tree Search Unifies Reasoning Acting and Planning
+  in Language Models'
+id: 231004406-language-agent-tree-search-unifies-reasoning-acting-and-planning-in-la-2
+tags:
+- deepread
+created: '2026-06-10T00:40:44.608943Z'
+source: https://ar5iv.labs.arxiv.org/html/2310.04406
+source_domain: ar5iv.labs.arxiv.org
+fetched_at: '2026-06-10T00:40:44.608803Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2310.04406] Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
+Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
+Andy Zhou
+University of Illinois at Urbana-Champaign
+AI@UIUC
+Kai Yan
+University of Illinois at Urbana-Champaign
+Michal Shlapentokh-Rothman
+University of Illinois at Urbana-Champaign
+Haohan Wang
+University of Illinois at Urbana-Champaign
+Yu-Xiong Wang
+University of Illinois at Urbana-Champaign
+Abstract
+While large language models (LLMs) have demonstrated impressive performance on a range of decision-making tasks, they rely on simple acting processes and fall short of broad deployment as autonomous agents. We introduce LATS (Language Agent Tree Search), a general framework that synergizes the capabilities of LLMs in planning, acting, and reasoning. Drawing inspiration from Monte Carlo tree search commonly used in model-based reinforcement learning, LATS employs LLMs as agents, value functions, and optimizers, repurposing their latent strengths for enhanced decision-making. What is crucial in this method is the use of an environment for external feedback, which offers a more deliberate and adaptive problem-solving mechanism that moves beyond the limitations of existing techniques. Our experimental evaluation across diverse domains, such as programming, HotPotQA, and WebShop, illustrates the applicability of LATS for decision-making while maintaining competitive reasoning performance. In particular, LATS achieves 94.4% for programming on HumanEval with GPT-4 and an average score of 75.9 for web browsing on WebShop with GPT-3.5, demonstrating the effectiveness and generality of our method.
+1
+Introduction
+General autonomous agents capable of reasoning and decision-making in a variety of environments
+(Wooldridge & Jennings,
+1995
+)
+have been of longstanding interest in the field of artificial intelligence. While this has traditionally been studied in reinforcement learning, the recent rise of large language models (LLMs)
+(Brown et al.,
+2020
+; Chowdhery et al.,
+2022
+; Touvron et al.,
+2023
+; OpenAI,
+2023
+)
+with strong reasoning and general adaptability offers an alternative paradigm. Not only have LLMs excelled on standard NLP tasks such as text summarization
+(Nallapati et al.,
+2016
+)
+or natural language inference
+(Bowman et al.,
+2015
+)
+, but they have been adapted to an increasingly diverse set of tasks that often require advanced common-sense reasoning or quantitative skills
+(Cobbe et al.,
+2021
+; Saparov & He,
+2022
+)
+. LLMs are also capable of performing in complex environments that involve knowledge and reasoning, such as web navigation
+(Yao et al.,
+2022
+; Deng et al.,
+2023
+)
+, tool-use
+(Schick et al.,
+2023
+)
+, or open-ended games
+(Fan et al.,
+2022
+)
+.
+Figure 1:
+An overview of LATS. LATS uses an external environment and self-reflection to improve reasoning and decision-making.
+Reasoning and acting abilities have also been improved by prompting techniques that augment LLMs with feedback or observations from an external environment
+(Yao et al.,
+2023b
+; Gao et al.,
+2022
+; Shinn et al.,
+2023
+)
+. This eliminates the need to rely entirely on the base abilities of the Language Model (LM), enhancing it through external tools or semantic feedback. Despite this strength, these methods are reflexive and fall short of humans’ deliberate and thoughtful decision-making characteristics to solve problems
+(Sloman,
+1996
+; Evans,
+2010
+)
+. In particular, such methods fail to consider multiple reasoning paths or to plan ahead. Recent search-guided LLM works
+(Xie et al.,
+2023
+; Yao et al.,
+2023a
+; Hao et al.,
+2023
+)
+address this issue by searching over multiple reasoning chains. While these methods enable planning, these methods operate in isolation and do not incorporate external feedback that can improve reasoning.
+To help address these issues, we propose LATS (Language Agent Tree Search), a general framework for decision-making and reasoning with language models. LATS unifies LM planning, acting, and reasoning strategies by expanding ReAct
+(Yao et al.,
+2023b
+)
+into a search over a combinatorial space of possible reasoning and acting steps. We adapt Monte Carlo tree search (MCTS) from model-based reinforcement learning
+(Silver et al.,
+2017
+; Anthony et al.,
+2017
+; Jiang et al.,
+2018
+)
+to language agents, repurposing a pretrained LLM as an agent, value function, and optimizer. Utilizing the strong natural language understanding and in-context learning ability of modern LMs, we use text as an interface between each component of the framework, allowing LATS to adapt planning to environmental conditions without additional training. To the best of our knowledge,
+LATS is the first framework that combines reasoning, acting, and planning to enhance LLMs
+. Notably, LATS doubles the performance of GPT-3.5 on HotPotQA
+(Yang et al.,
+2018
+)
+over ReAct
+(Yao et al.,
+2023b
+)
+and raises the average score by
+22.1
+22.1
+22.1
+on WebShop
+(Yao et al.,
+2022
+)
+. When used with GPT-4, LATS achieves a
+94.4
+94.4
+94.4
+Pass@1 rate for programming on HumanEval
+(Chen et al.,
+2021
+)
+, setting the state of the art. To summarize, our
+contributions
+are the following:
+•
+We introduce an LM-based Monte Carlo tree search variant to deliberately construct the best trajectory from sampled actions, enabling more flexible and adaptive problem-solving compared to reflexive prompting methods. This is guided by heuristics from the LM.
+•
+By integrating external feedback and self-reflection, LATS enhances model sensibility and enables agents to learn from experience, surpassing reasoning-based search methods.
+•
+Through experiments across diverse domains like programming, interactive QA, and web navigation, we demonstrate the versatility of LATS in harnessing LLMs for autonomous reasoning and decision-making.
+2
+Related Work
+Approach
+Reasoning
+Acting
+Planning
+Self
+External
+Reflection
+Memory
+CoT
+(Wei et al.,
+2022
+)
+✓
+×
+\times
+×
+\times
+×
+\times
+×
+\times
+ReAct
+(Yao et al.,
+2023b
+)
+✓
+✓
+×
+\times
+×
+\times
+×
+\times
+ToT
+(Yao et al.,
+2023a
+)
+✓
+×
+\times
+✓
+✓
+✓
+RAP
+(Hao et al.,
+2023
+)
+✓
+×
+\times
+✓
+×
+\times
+✓
+Self-Refine
+(Madaan et al.,
+2023
+)
+✓
+×
+\times
+×
+\times
+✓
+×
+\times
+Beam Search
+(Xie et al.,
+2023
+)
+✓
+×
+\times
+×
+\times
+✓
+×
+\times
+Reflexion
+(Shinn et al.,
+2023
+)
+✓
+✓
+×
+\times
+✓
+✓
+LATS (Ours)
+✓
+✓
+✓
+✓
+✓
+Table 1:
+A summary of related work on reasoning, acting, and planning. LATS is the first work incorporating designs from all three domains, allowing use in all corresponding tasks. We refer to planning as the use of a search algorithm, self-reflection as the use of LM-generated feedback, and external memory as storaging past text context for future updates of solution.
+a) Tree-of-Thoughts
+b) Reasoning via Planning
+c) Language Agent Tree Search
+Figure 2:
+An overview of the differences between LATS and recently proposed LM search algorithms ToT
+(Yao et al.,
+2023a
+)
+and RAP
+(Hao et al.,
+2023
+)
+. LATS leverages environmental feedback and self-reflection to further adapt search and improve performance.
+LLMs for reasoning.
+For LLMs, reasoning typically involves decomposing complex inputs into sequential intermediate steps towards a final answer
+(Cobbe et al.,
+2021
+)
+, demonstrated with Chain-of-Thought (CoT) prompting
+(Wei et al.,
+2022
+)
+and its variants
+(Wei et al.,
+2022
+; Kojima et al.,
+2022
+; Wang et al.,
+2022
+)
+. However, these methods, which create chains autoregressively in a single step, often suffer from error propagation as the number of steps increases
+(Guo et al.,
+2018
+; Chen et al.,
+2022b
+)
+due to compound errors. Various advancements aim to mitigate this issue; some approaches, such as Self-Consistency
+(Wang et al.,
+2022
+)
+, employ majority voting over sampled chains, while others focus on multi-step decomposition, such as least-to-most prompting
+(Zhou et al.,
+2022
+)
+, or use of external tools such as a scratchpad
+(Nye et al.,
+2021
+)
+or compiler
+(Gao et al.,
+2022
+)
+. Recently, CoT has been improved with search algorithms
+(Yao et al.,
+2023a
+; Hao et al.,
+2023
+; Besta et al.,
+2023
+)
+that can sample trajectories more effectively. Tree-of-thought (ToT) prompting
+(Yao et al.,
+2023a
+)
+uses DFS or BFS-based search guided by an LM-generated heuristic while Reasoning via Planning (RAP)
+(Hao et al.,
+2023
+)
+uses MCTS with rollouts simulated by the LM. However, they rely solely on LM internal knowledge and cannot adapt to useful external feedback.
+LLMs for acting.
+The strong reasoning and common-sense abilities of LLMs have also been adapted for decision-making or acting tasks as a policy model in interactive environments. In the realm of robotics LLMs have been employed as high-level controllers of control policies
+(Ahn et al.,
+2022
+; Huang et al.,
+2022
+; Driess et al.,
+2023
+)
+. Similar work
+(Baker et al.,
+2022
+; Wang et al.,
+2023
+; Zhu et al.,
+2023
+)
+has also adapted LLM agents to complex multimodal games such as Minecraft
+(Guss et al.,
+2019
+; Fan et al.,
+2022
+)
+. LLMs are particularly useful in text-based environments
+(Liu et al.,
+2018
+; Shridhar et al.,
+2020
+; Liu et al.,
+2023
+)
+, where acting-based prompting techniques such as ReAct
+(Yao et al.,
+2023b
+)
+have seen success. Similar to CoT, ReAct is limited by its simplicity and cannot effectively adapt to environment conditions. Many extensions have been proposed to address this, including Self-refine
+(Madaan et al.,
+2023
+)
+and Reflexion
+(Shinn et al.,
+2023
+; Yao et al.,
+2023c
+)
+, which uses self-reflection to enhance reasoning and decision-making, and AdaPlanner
+(Sun et al.,
+2023
+)
+, which incorporates both positive and negative environmental feedback. However these methods focus on refining an individual plan or trajectory and do not consider alternative choices at each step. In addition, recent work
+(Huang et al.,
+2023
+)
+has suggested LLMs cannot self-correct their internal reasoning, making it critical to use external feedback. Alternatively to pure decision-making environments, the reasoning and practical abilities of LLMs have been enhanced by access to external tools, such as APIs, search engines, calculators, or other models
+(Schick et al.,
+2023
+; Shen et al.,
+2023
+; Surís et al.,
+2023
+)
+. Contrary to reasoning-based approaches, these methods have not been improved with planning, limiting their effectiveness. We summarize them in Tab.
+1
+.
+Tree-based search.
+Tree-based search, where multiple branches of outcomes are explored during search, is widely used in many planning algorithms
+(Świechowski et al.,
+2023
+; LaValle et al.,
+2001
+)
+and Reinforcement Learning (RL)
+(Hafner et al.,
+2019
+; Du et al.,
+2023
+; Wu et al.,
+2023
+)
+algorithms for its good exploration-exploitation trade-off. Though tree-based search requires an environment model that can expand from arbitrary state
+(Vodopivec et al.,
+2017
+)
+, which often requires extra training in RL
+(Hafner et al.,
+2023
+)
+, such problem does not exist for LM tasks as we can conveniently backup to any state by setting the input to be the context and corresponding previous output by the LM. Thus, we work on the tree-based framework and use MCTS
+(Świechowski et al.,
+2023
+)
+to fully release the potential of LMs, while avoiding the cost of training a value function over language descriptions by leveraging the in-context learning
+(Brown et al.,
+2020
+)
+abilities of LLMs.
+3
+Preliminaries
+3.1
+Problem Setting and Prompting
+Before describing LATS, we first define our problem and outline a few established methods that leverage large language models for reasoning or decision-making. In LM reasoning or decision making, we are given an input
+x
+𝑥
+x
+in natural language and a pretrained language model
+p
+θ
+​
+(
+x
+)
+subscript
+𝑝
+𝜃
+𝑥
+p_{\theta}(x)
+parameterized by
+θ
+𝜃
+\theta
+; our goal is to generate a final output
+y
+∼
+p
+θ
+​
+(
+x
+)
+similar-to
+𝑦
+subscript
+𝑝
+𝜃
+𝑥
+y\sim p_{\theta}(x)
+corresponding to the answer (reasoning) or completes the task (decision-making). Both
+x
+𝑥
+x
+and
+y
+𝑦
+y
+are language
+sequences
+, which are comprised of a list of
+tokens
+(the basic elements of natural language, often words), denoted as
+x
+=
+(
+x
+​
+[
+1
+]
+,
+…
+,
+x
+​
+[
+n
+]
+)
+𝑥
+𝑥
+delimited-[]
+1
+…
+𝑥
+delimited-[]
+𝑛
+x=(x[1],\dots,x[n])
+and
+y
+=
+(
+y
+​
+[
+1
+]
+,
+…
+,
+y
+​
+[
+n
+]
+)
+𝑦
+𝑦
+delimited-[]
+1
+…
+𝑦
+delimited-[]
+𝑛
+y=(y[1],\dots,y[n])
+. The LM decodes text autoregressively, i.e., without other inputs, the probability for an LM to generate a sequence
+x
+𝑥
+x
+is given by
+p
+θ
+​
+(
+x
+)
+=
+∏
+i
+=
+1
+n
+p
+θ
+​
+(
+x
+​
+[
+i
+]
+|
+x
+​
+[
+1
+​
+…
+​
+i
+−
+1
+]
+)
+subscript
+𝑝
+𝜃
+𝑥
+superscript
+subscript
+product
+𝑖
+1
+𝑛
+subscript
+𝑝
+𝜃
+conditional
+𝑥
+delimited-[]
+𝑖
+𝑥
+delimited-[]
+1
+…
+𝑖
+1
+p_{\theta}(x)=\prod_{i=1}^{n}p_{\theta}(x[i]|x[1\dots i-1])
+. Usually, to improve the LM,
+prompts
+are provided along with the input
+x
+𝑥
+x
+, which are specific instructions or few-shot input-output examples. We denote the generic process where an input
+x
+𝑥
+x
+is transformed into an output
+y
+𝑦
+y
+by LM:
+y
+∼
+p
+θ
+​
+(
+y
+|
+prompt
+I
+​
+O
+​
+(
+x
+)
+)
+similar-to
+𝑦
+subscript
+𝑝
+𝜃
+conditional
+𝑦
+subscript
+prompt
+𝐼
+𝑂
+𝑥
+y\sim p_{\theta}(y|\texttt{prompt}_{IO}(x))
+, where
+prompt
+I
+​
+O
+​
+(
+x
+)
+subscript
+prompt
+𝐼
+𝑂
+𝑥
+\texttt{prompt}_{IO}(x)
+denotes the input
+x
+𝑥
+x
+.
+Chain-of-thought (CoT) Prompting
+(Wei et al.,
+2022
+)
+was introduced to cater to scenarios where direct mapping from
+x
+𝑥
+x
+to
+y
+𝑦
+y
+is intricate, such as when
+x
+𝑥
+x
+is from a mathematical query or challenging question. This method hinges on creating
+thoughts
+z
+1
+,
+…
+,
+z
+n
+subscript
+𝑧
+1
+…
+subscript
+𝑧
+𝑛
+z_{1},\dots,z_{n}
+that act as stepping stones between
+x
+𝑥
+x
+and
+y
+𝑦
+y
+; each thought
+z
+i
+subscript
+𝑧
+𝑖
+z_{i}
+is a language sequence. To employ CoT prompting, thoughts are extracted sequentially as
+z
+i
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+z
+i
+|
+x
+,
+z
+1
+​
+⋯
+​
+i
+−
+1
+)
+similar-to
+subscript
+𝑧
+𝑖
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+subscript
+𝑧
+𝑖
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑖
+1
+z_{i}\sim p_{\theta}^{CoT}(z_{i}|x,z_{1\cdots i-1})
+, with the final output being
+y
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+y
+|
+x
+,
+z
+1
+​
+⋯
+​
+n
+)
+similar-to
+𝑦
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+𝑦
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑛
+y\sim p_{\theta}^{CoT}(y|x,z_{1\cdots n})
+.
+Tree-of-thought (ToT) Prompting
+(Yao et al.,
+2023a
+)
+extends CoT prompting by exploring multiple reasoning paths over thoughts. It frames problems as a search over a tree where each node
+s
+=
+[
+x
+,
+z
+1
+⋅
+i
+]
+𝑠
+𝑥
+subscript
+𝑧
+⋅
+1
+𝑖
+s=[x,z_{1\cdot i}]
+represents a partial solution state comprising the original input
+x
+𝑥
+x
+and thought sequence
+z
+1
+​
+⋯
+​
+i
+subscript
+𝑧
+1
+⋯
+𝑖
+z_{1\cdots i}
+. Thoughts
+z
+i
+subscript
+𝑧
+𝑖
+z_{i}
+are generated by proposal or sampling with CoT
+z
+i
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+z
+i
+|
+x
+,
+z
+1
+​
+⋯
+​
+i
+−
+1
+)
+similar-to
+subscript
+𝑧
+𝑖
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+subscript
+𝑧
+𝑖
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑖
+1
+z_{i}\sim p_{\theta}^{CoT}(z_{i}|x,z_{1\cdots i-1})
+. Deliberate search algorithms like breadth-first or depth-first search are used to systematically explore the tree, guided by heuristics based on language model evaluations
+V
+​
+(
+s
+)
+𝑉
+𝑠
+V(s)
+of each state.
+Reasoning via Planning
+(RAP)
+(Hao et al.,
+2023
+)
+is similar to ToT, except that MCTS is used over DFS or BFS. Heuristics are designed from an LM, such as the likelihood or confidence of an action, and the LM is used as a world model to predict subsequent states during the simulation step.
+ReAct
+(Yao et al.,
+2023b
+)
+extends language models to tasks where the mapping from
+x
+𝑥
+x
+to
+y
+𝑦
+y
+is enhanced by or requires interactions with an external environment, such as a game or API. This technique constructs an action space
+A
+^
+=
+A
+∪
+Z
+^
+𝐴
+𝐴
+𝑍
+\hat{A}=A\cup Z
+that adds permissible actions
+a
+𝑎
+a
+to the reasoning traces
+z
+𝑧
+z
+from CoT. Observations
+o
+𝑜
+o
+from the environment are used to improve both reasoning and acting. To solve problems with ReAct, after each observation, actions are generated from
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+sequentially as
+a
+i
+∼
+p
+θ
+R
+​
+e
+​
+A
+​
+c
+​
+t
+​
+(
+a
+i
+|
+x
+,
+o
+1
+​
+⋯
+​
+i
+−
+1
+,
+a
+1
+​
+⋯
+​
+i
+−
+1
+)
+similar-to
+subscript
+𝑎
+𝑖
+superscript
+subscript
+𝑝
+𝜃
+𝑅
+𝑒
+𝐴
+𝑐
+𝑡
+conditional
+subscript
+𝑎
+𝑖
+𝑥
+subscript
+𝑜
+1
+⋯
+𝑖
+1
+subscript
+𝑎
+1
+⋯
+𝑖
+1
+a_{i}\sim p_{\theta}^{ReAct}(a_{i}|x,o_{1\cdots i-1},a_{1\cdots i-1})
+, with the final output being
+y
+∼
+p
+θ
+R
+​
+e
+​
+A
+​
+c
+​
+t
+​
+(
+y
+|
+x
+,
+o
+1
+​
+⋯
+​
+n
+,
+a
+1
+​
+⋯
+​
+n
+)
+similar-to
+𝑦
+superscript
+subscript
+𝑝
+𝜃
+𝑅
+𝑒
+𝐴
+𝑐
+𝑡
+conditional
+𝑦
+𝑥
+subscript
+𝑜
+1
+⋯
+𝑛
+subscript
+𝑎
+1
+⋯
+𝑛
+y\sim p_{\theta}^{ReAct}(y~{}|~{}x,o_{1\cdots n},a_{1\cdots n})
+.
+While the previously described prompting techniques improve LM performance on reasoning tasks, they falter on difficult tasks that involve multifaceted decision-making due to several shortcomings: 1)
+Flexibility
+: Base prompting methods (CoT or ReAct) autoregressively sample from the LM, neglecting potential alternative continuations from specific states. 2)
+Sensibility
+: Reasoning-based methods (CoT, RAP, or ToT) rely solely on the internal representations of the LM and cannot consider external observations. This dependency risks fact hallucination and error propagation while setting a performance ceiling. 3)
+Adaptability
+: Current planning frameworks (RAP or ToT) use simple search algorithms such as BFS or cannot leverage environmental feedback to improve planning. Additionally, the agent is static and cannot reuse previous experience or learn from trial and error. While RAP also adopts MCTS, it is constrained to tasks where the LM can become a world model and accurately predict states. These shortcomings limit the ability of LMs to be deployed as general problem-solving agents and form the motivation for LATS.
+3.2
+Monte-Carlo Tree Search (MCTS)
+Monte-Carlo Tree Search (MCTS) is a heuristic search algorithm that is proved successful on many decision-making environments such as Atari
+(Ye et al.,
+2021
+)
+and Go
+(Silver et al.,
+2016
+)
+. MCTS builds a decision tree where every node in the tree is a state and edge is an action. MCTS runs for
+k
+𝑘
+k
+episodes; for each episode, it starts from the root (i.e., initial state) and iteratively conducts two steps to expand the tree: 1)
+Expansion
+, where multiple children states
+s
+𝑠
+s
+are explored from the current parent state
+p
+𝑝
+p
+by sampling
+n
+𝑛
+n
+actions, and 2)
+Selection
+, where the children with the highest UCT
+(Upper Confidence bounds applied to Trees)
+(Kocsis & Szepesvári,
+2006
+)
+value is selected by the next iteration. The UCT of a child state
+s
+𝑠
+s
+is calculated as follows:
+U
+​
+C
+​
+T
+​
+(
+s
+)
+=
+V
+​
+(
+s
+)
++
+w
+​
+ln
+⁡
+N
+​
+(
+p
+)
+N
+​
+(
+s
+)
+,
+𝑈
+𝐶
+𝑇
+𝑠
+𝑉
+𝑠
+𝑤
+𝑁
+𝑝
+𝑁
+𝑠
+UCT(s)=V(s)+w\sqrt{\frac{\ln N(p)}{N(s)}},
+(1)
+where
+N
+​
+(
+s
+)
+𝑁
+𝑠
+N(s)
+is the number of visits to a node
+s
+𝑠
+s
+,
+V
+​
+(
+s
+)
+𝑉
+𝑠
+V(s)
+is the value function (expected return) from the subtree of
+s
+𝑠
+s
+,
+w
+𝑤
+w
+is the exploration weight, and
+p
+𝑝
+p
+is the parent node of
+s
+𝑠
+s
+. The child node with the highest UCT value is selected for expansion in the next iteration. When the end of an episode is reached, a
+backpropagation
+is carried out: the return
+r
+𝑟
+r
+is used for updating every
+V
+​
+(
+s
+)
+𝑉
+𝑠
+V(s)
+along the path
+with the formula
+V
+​
+(
+s
+)
+=
+V
+old
+​
+(
+s
+)
+​
+(
+N
+​
+(
+s
+)
+−
+1
+)
++
+r
+N
+​
+(
+s
+)
+𝑉
+𝑠
+subscript
+𝑉
+old
+𝑠
+𝑁
+𝑠
+1
+𝑟
+𝑁
+𝑠
+V(s)=\frac{V_{\text{old}}(s)(N(s)-1)+r}{N(s)}
+, where
+V
+old
+​
+(
+s
+)
+subscript
+𝑉
+old
+𝑠
+V_{\text{old}}(s)
+is the old value function. Normally, the major shortcoming of MCTS is that it requires an environment model to undo previous steps and form a searching tree, which is often a strong assumption. However, such a limitation does not exist for LMs, as we can conveniently reset to any step by simply copy-pasting historical text input. Such a special property is the key motivation of our work.
+4
+Unifying Planning, Reasoning, and Acting
+4.1
+LM Agent
+LATS supports sequential reasoning or decision-making tasks on the basis of ReAct. At time step
+t
+𝑡
+t
+, an agent receives an observation
+o
+t
+∈
+O
+subscript
+𝑜
+𝑡
+𝑂
+o_{t}\in O
+from the environment and takes an action
+a
+t
+∈
+A
+subscript
+𝑎
+𝑡
+𝐴
+a_{t}\in A
+following some policy
+π
+​
+(
+a
+t
+|
+x
+,
+o
+1
+​
+⋯
+​
+i
+−
+1
+,
+a
+1
+​
+⋯
+​
+i
+−
+1
+)
+𝜋
+conditional
+subscript
+𝑎
+𝑡
+𝑥
+subscript
+𝑜
+1
+⋯
+𝑖
+1
+subscript
+𝑎
+1
+⋯
+𝑖
+1
+\pi(a_{t}|x,o_{1\cdots i-1},a_{1\cdots i-1})
+, where
+x
+𝑥
+x
+consists of the task instruction and a number of few-shot examples. We initialize the agent with
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+to leverage the useful language representations of an LM as a base decision-maker. We follow the ReAct instantiation in which the action space
+A
+^
+=
+A
+∪
+Z
+^
+𝐴
+𝐴
+𝑍
+\hat{A}=A\cup Z
+consists of both the space of permissible actions
+A
+𝐴
+A
+and language space of reasoning traces
+Z
+𝑍
+Z
+. Actions directly affect the environment and result in observation, while thoughts are used to formalize decisions by organizing information, planning future actions, or injecting internal knowledge. The exact instantiation of the action space depends on the particular environment; for decision-making tasks actions might consist of commands on a website while for reasoning tasks the action space might be limited to a few external tools or APIs.
+Instead of greedily decoding one trajectory or solution, we sample
+n
+𝑛
+n
+actions from
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+using the current state. This is based on the intuition that for complex decision-making tasks, there is likely to be a range of potential trajectories or reasoning paths that are correct
+(Evans,
+2010
+)
+. Sampling a diverse set of candidates at each step mitigates the stochastic nature of LM text generation and enables greater exploration in both the decision-making and reasoning space. We wrap
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+within our proposed search algorithm to deliberately construct the best trajectory from sampled actions.
+4.2
+LATS
+Figure 3:
+An overview of the six operations of LATS. A node is
+selected
+,
+expanded
+,
+evaluated
+, then
+simulated
+until a terminal node is reached, then the resulting value is
+backpropagated
+. If the trajectory fails, a
+reflection
+is generated and used as additional context for future trials. These operations are performed in succession until the budget is reached or task is successful.
+The main component of LATS is a search algorithm that controls the overall problem-solving process with deliberate planning. To find the most promising trajectory and systemically balance exploration with exploitation, we adopt a variant of Monte Carlo Tree Search (MCTS) that frames decision-making as a tree search, in which each node
+s
+=
+[
+x
+,
+a
+1
+​
+⋯
+​
+i
+,
+o
+1
+​
+⋯
+​
+i
+]
+𝑠
+𝑥
+subscript
+𝑎
+1
+⋯
+𝑖
+subscript
+𝑜
+1
+⋯
+𝑖
+s=[x,a_{1\cdots i},o_{1\cdots i}]
+represents a state comprising the original input
+x
+𝑥
+x
+, action sequence
+a
+1
+⋅
+i
+subscript
+𝑎
+⋅
+1
+𝑖
+a_{1\cdot i}
+, and observation sequence
+o
+1
+⋅
+i
+subscript
+𝑜
+⋅
+1
+𝑖
+o_{1\cdot i}
+.
+To adapt MCTS for language agents, LATS repurposes
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+as an agent, state evaluator, and feedback generator, leveraging the useful language priors of modern LMs to facilitate planning. While standard MCTS and RAP
+Hao et al. (
+2023
+)
+rely on internal dynamics models to facilitate simulation, LATS is model-free and uses environment interaction. LATS consists of a series of operations,
+selection, expansion, evaluation, simulation, backpropagation, and reflection
+, performed in succession until the task is successfully completed or a computational limit is reached. The full psuedocode of LATS can be found in Sec.
+A
+in the Appendix.
+Selection.
+In the first operation, the algorithm identifies a segment of the current tree most suitable for subsequent expansion. Starting from the root node, denoted as the initial state
+s
+0
+subscript
+𝑠
+0
+s_{0}
+, a child node is selected at each tree level until a leaf node is reached. To balance exploration and exploitation, we use the UCT algorithm as shown in Eq.
+1
+.
+Expansion.
+After selecting a node, the second operation expands the tree by sampling
+n
+𝑛
+n
+actions from
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+, as described in the prior section. The environment receives each action and returns corresponding feedback as an observation. This results in
+n
+𝑛
+n
+new child nodes added to the tree. This tree is stored in an external long-term memory structure.
+Evaluation.
+The third operation assigns a scalar value to each new child node to be used for selection and backpropagation. This value effectively quantifies the agent’s progress in task completion, serving as a heuristic to steer the search algorithm towards the most promising regions of the tree. Following
+Yao et al. (
+2023a
+)
+we repurpose
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+into a value function by prompting it to reason about a given state. To obtain a scalar value, we instruct
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+to end its reasoning trace with a score indicating the correctness of the trajectory. This method offers enhanced flexibility over programmed heuristics
+(Campbell et al.,
+2002
+)
+and greater efficiency than learned heuristics
+(Silver et al.,
+2017
+)
+.
+Simulation.
+The fourth operation expands the currently selected node until a terminal state is reached. At each depth level we sample and evaluate nodes with the same operations, but prioritize nodes of highest value. Reaching a terminal state provides objective feedback on the correctness of a trajectory. If the task is completed successfully, then LATS terminates the search. If the solution is partially successful or unsuccessful, then we perform two additional operations as described below.
+Backpropagation.
+This operation updates the values of the tree based on the outcome of a trajectory. For each node
+s
+0
+,
+s
+1
+,
+…
+,
+s
+n
+subscript
+𝑠
+0
+subscript
+𝑠
+1
+…
+subscript
+𝑠
+𝑛
+s_{0},s_{1},\dots,s_{n}
+in the trajectory from root (initial state
+s
+0
+subscript
+𝑠
+0
+s_{0}
+) of the searching tree to leaf (terminal state
+s
+n
+subscript
+𝑠
+𝑛
+s_{n}
+), its value is updated to reflect the outcome of the simulation by
+N
+​
+(
+s
+i
+)
+=
+N
+old
+​
+(
+s
+i
+)
++
+1
+𝑁
+subscript
+𝑠
+𝑖
+subscript
+𝑁
+old
+subscript
+𝑠
+𝑖
+1
+N(s_{i})=N_{\text{old}}(s_{i})+1
+and
+V
+​
+(
+s
+i
+)
+=
+r
++
+N
+old
+​
+(
+s
+i
+)
+​
+V
+old
+​
+(
+s
+i
+)
+N
+​
+(
+s
+i
+)
+𝑉
+subscript
+𝑠
+𝑖
+𝑟
+subscript
+𝑁
+old
+subscript
+𝑠
+𝑖
+subscript
+𝑉
+old
+subscript
+𝑠
+𝑖
+𝑁
+subscript
+𝑠
+𝑖
+V(s_{i})=\frac{r+N_{\text{old}}(s_{i})V_{\text{old}}(s_{i})}{N(s_{i})}
+, where
+r
+𝑟
+r
+is the return and
+N
+old
+,
+V
+old
+subscript
+𝑁
+old
+subscript
+𝑉
+old
+N_{\text{old}},V_{\text{old}}
+are the old number of visits and value function. These updated values are used in the UCT formula (Eq.
+1
+) to guide the selection of the next node for exploration.
+Reflection.
+In addition to the environmental feedback, we also leverage
+self-reflection
+to further refine the decision-making process
+(Shinn et al.,
+2023
+; Madaan et al.,
+2023
+)
+. Upon encountering an unsuccessful terminal node,
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+is prompted with the trajectory and final reward to provide a verbal self-reflection that summarizes the errors in the reasoning or acting process and proposes superior alternatives. We store both failed trajectories and corresponding reflections in the memory. In subsequent iterations, these are integrated as additional context to the agent and value function, refining both through in-context learning. This imparts a semantic gradient signal more useful than a scalar value, enabling the agent to learn from trial and error without the cost of expensive optimization processes such as reinforcement learning.
+Conceptually, LATS has the following advantages as a general framework for reasoning and decision-making with LM agents.
+(1)
+Generality
+: LATS supports both reasoning and decision-making tasks by defining a shared space of thoughts and actions. (2)
+Deliberate
+: The use of MCTS and LM value function ensures a principled search that selects options with high value while exploring promising alternatives. (3)
+Adaptability
+: LATS is designed around the use of external feedback through observations and self-reflection, enabling greater adaptation during problem-solving. (4)
+Flexibility
+: LATS can accommodate different scenarios, environments, and resource stipulations by modifying state design and tree dimensions. (5)
+Modularity
+: The base LM agent, reflection generator, and value function can be independently altered and adapted to individual LM properties.
+5
+Experiments
+To demonstrate the general applicability of LATS, we evaluate our method on a variety of decision-making domains that requires both reasoning and acting ability: programming
+(Chen et al.,
+2021
+; Austin et al.,
+2021
+)
+, HotPotQA
+(Yang et al.,
+2018
+)
+, and WebShop
+(Yao et al.,
+2022
+)
+.
+5.1
+HotPotQA
+For a task that can be approached with both reasoning-based and acting-based strategies, we consider HotPotQA
+(Yang et al.,
+2018
+)
+, a multi-hop question-answering benchmark that requires retrieval over two or more Wikipedia passages. For the action space, in addition to LM thoughts we follow the setup from
+Yao et al. (
+2023b
+)
+, which provides the agent with API calls to search and lookup information. The output of these API calls and self-generated reflections form the observation space. We use a subset of 100 questions and three few-shot examples for each method. For ToT, we use DFS as the base search algorithm and scoring with the LM as the heuristic. For all methods that involve sampling, including LATS, we sample
+k
+=
+50
+𝑘
+50
+k=50
+trajectories. More details and prompts can be found in Sec.
+D
+and Sec.
+E
+in the Appendix.
+We evaluate internal reasoning strategies by removing actions and observations from the context, corresponding to CoT
+(Wei et al.,
+2022
+)
+and its variants, CoT-SC
+(Wang et al.,
+2022
+)
+, ToT
+(Yao et al.,
+2023a
+)
+, and RAP
+(Hao et al.,
+2023
+)
+. These methods rely solely on the agent’s existing knowledge to answer the question. We also consider acting-based methods ReAct, Reflexion, and LATS, which augment the agent with the interactive API environment and primarily evaluate its information retrieval abilities. While LATS is designed for scenarios where external feedback can enhance reasoning, we also implement a reasoning-only version with CoT as the base prompt. We also combine internal and external reasoning in LATS by first prompting with a CoT-based prompt, then switching to a ReAct-based prompt upon failure. This is closer to how humans might approach this task, by using tools to lookup additional information only when the answer is not already known.
+Prompt Method
+HotpotQA (EM)
+I/O
+0.32
+CoT
+(Wei et al.,
+2022
+)
+0.34
+CoT - SC
+(Wang et al.,
+2022
+)
+0.38
+ToT
+(Yao et al.,
+2023a
+)
+0.55
+RAP
+(Hao et al.,
+2023
+)
+0.60
+RAP (n = 10)
+0.60
+LATS (CoT)
+0.60
+Prompt Method
+HotpotQA (EM)
+ReAct
+(Yao et al.,
+2023b
+)
+0.32
+ReAct (best of k)
+0.38
+Reflexion
+(Shinn et al.,
+2023
+)
+0.51
+LATS
+0.61
+LATS (n = 3)
+0.56
+LATS (n = 10)
+0.64
+LATS (CoT + ReAct)
+0.71
+Table 2:
+GPT-3.5 reasoning-based prompting (left) and acting-based prompting (right) results on HotpotQA. LATS achieves the highest exact match (EM) for acting and is competitive on reasoning. Unless otherwise specified, we sample
+n
+=
+5
+𝑛
+5
+n=5
+nodes during expansion and
+k
+=
+50
+𝑘
+50
+k=50
+trajectories.
+Results.
+We observe in Tab.
+2
+that both internal reasoning and external retrieval strategies perform well on HotPotQA. Due to their large-scale training corpus, modern LLMs already encode factual knowledge and can often directly answer the question correctly. While CoT can slightly enhance performance on questions requiring reasoning, larger gains are observed with search methods ToT and RAP, which can sample and explore more outputs. We observe similar results for acting-based methods. LATS surpasses ReAct, even when sampling the same number of trajectories, by expanding more nodes with principled search (see Fig.
+5
+in Appendix
+D
+for a qualitative sample). This is demonstrated when modifying
+n
+𝑛
+n
+, the number of nodes expanded during each iteration. Increasing
+n
+𝑛
+n
+can consistently improve performance, although at greater computational and inference costs. LATS is also competitive to RAP on internal reasoning but performs worse than acting. Combining internal and external reasoning in LATS results in the highest performance, indicating the importance of external feedback in augmenting reasoning even in tasks the base LM can already perform.
+5.2
+Programming
+Prompt Method
+Model
+Pass@1
+CoT
+(Wei et al.,
+2022
+)
+GPT-3.5
+46.9
+ReAct
+(Yao et al.,
+2023b
+)
+GPT-3.5
+56.9
+Reflexion
+(Shinn et al.,
+2023
+)
+GPT-3.5
+68.1
+ToT
+(Yao et al.,
+2023a
+)
+GPT-3.5
+54.4
+RAP
+(Hao et al.,
+2023
+)
+GPT-3.5
+63.1
+LATS (Ours)
+GPT-3.5
+83.8
+I/O
+GPT-4
+80.1
+Reflexion
+GPT-4
+91.0
+LATS
+GPT-4
+94.4
+Prompt Method
+Pass@1
+CoT
+(Wei et al.,
+2022
+)
+54.9
+ReAct
+(Wei et al.,
+2022
+)
+67.0
+Reflexion
+(Shinn et al.,
+2023
+)
+70.0
+ToT
+(Yao et al.,
+2023a
+)
+65.8
+RAP
+(Hao et al.,
+2023
+)
+71.4
+LATS (Ours)
+81.1
+Table 3:
+GPT-3.5 and GPT-4 Pass@1 accuracy on HumanEval
+(Chen et al.,
+2021
+)
+and MBPP
+(Austin et al.,
+2021
+)
+. Prompting with LATS achieves the highest performance. We sample 5 solutions during expansion for
+8
+iterations.
+To demonstrate the importance of external observations for complex reasoning tasks, we evaluate the baselines and LATS on programming with Humaneval
+(Chen et al.,
+2021
+)
+and MBPP
+(Austin et al.,
+2021
+)
+. Both datasets measure the correctness of synthesized programs in Python from natural language docstrings. We use individual solutions as the action space and test suite and compiler feedback as the external observation. We follow
+Chen et al. (
+2022a
+)
+and use an LLM to generate a synthetic test suite of syntactically valid “assert” statements for each question. For each step, the solution is evaluated on this test suite, and the results including successful and failed tests and compiler output, are added to the context as an observation. We use the same test suite for Reflexion.
+For this task, the reasoning and acting baselines share an action space, but acting methods are able to incorporate observations as additional context. For LATS, since each action corresponds to a complete solution, we skip the simulation step of LATS and directly use the percentage of passed tests as the backpropagated reward. We use
+k
+=
+8
+𝑘
+8
+k=8
+iterations, set the number of generated tests at
+4
+4
+4
+, and sample
+n
+=
+5
+𝑛
+5
+n=5
+solutions during expansion. After the search is completed, we select the solution with the highest value and evaluate it on the real test suite for the pass@1 accuracy evaluation. More details and prompts can be found in Sec.
+D
+and Sec.
+F
+in the Appendix.
+Results.
+We find in Tab
+3
+that both search and semantic feedback are crucial for better performance. Despite not using observations, ToT and RAP are competitive with Reflexion. LATS has the highest performance on both datasets. Since RAP uses a similar search algorithm as LATS, this reveals the importance of external feedback for difficult reasoning tasks such as programming. With GPT-4, using LATS sets the state of the art for HumanEval, showing LATS can be used with more advanced LLMs for higher performance.
+5.3
+Webshop
+For a complex decision-making environment with practical applications, we consider WebShop
+(Yao et al.,
+2022
+)
+, an online shopping environment composed of a website with 1.18M real-world products and 12k human instructions. Agents must navigate a website through a variety of commands to purchase an item matching a user specification. We use the preconstructed action space of search and click commands and browser feedback and reflections for the observation. The performance is gauged using two metrics: an average score, reflecting the percentage of user-specified attributes met by the selected product, and a success rate, indicating the frequency with which the chosen product fulfills all given conditions. We compare against acting-based prompting methods and RL-based approaches. We evaluate on 50 instructions, expand
+n
+=
+5
+𝑛
+5
+n=5
+children for LATS, and set
+k
+=
+30
+𝑘
+30
+k=30
+for LATS, ReAct best of
+k
+𝑘
+k
+, and Reflexion. More details and prompts are in Appendix
+D
+and
+G
+.
+Results.
+We find in Tab.
+5
+that GPT-3.5 with ReAct is competitive to imitation learning, and can exceed reinforcement learning techniques with stronger prompting strategies. Sampling
+k
+=
+30
+𝑘
+30
+k=30
+trajectories with ReAct and Reflexion results in a similar performance, suggesting the semantic feedback is not as helpful in complex environments like WebShop. Indeed like in
+Shinn et al. (
+2023
+)
+, we find that generated reflections are often generic and do not provide useful feedback, resulting in a tendency for the agent to become stuck in local minima. However, using LATS indeed results in a noticeable improvement, indicating a more effective exploration for the same number of iterations.
+5.4
+Additional Observations
+Method
+Score
+SR
+ReAct
+(Yao et al.,
+2023b
+)
+53.8
+28.0
+ReAct (best of k)
+59.1
+32.0
+Reflexion
+(Shinn et al.,
+2023
+)
+64.2
+35.0
+LATS
+75.9
+38.0
+IL
+59.9
+29.1
+IL+RL
+62.4
+28.7
+Fine-tuning
+(Furuta et al.,
+2023
+)
+67.5
+45.0
+Expert
+82.1
+59.6
+Table 4:
+Score and success rate (SR) on Webshop. Table is separated into prompting, RL-based training, and human performance. For the same number of iterations, LATS improves both score and success rate, and surpasses RL-based training. IL/IL+RL taken from
+Yao et al. (
+2022
+)
+.
+Prompt Method
+HotPotQA (EM)
+ToT (ReAct)
+0.39
+RAP (ReAct)
+0.54
+LATS (No LM Heuristic)
+0.37
+LATS (DFS)
+0.42
+LATS (No Reflection)
+0.56
+LATS
+0.61
+Table 5:
+Ablation results on LATS and baseline variants in HotPotQA; we use ReAct as the base prompt and sample
+n
+=
+5
+𝑛
+5
+n=5
+children and
+k
+=
+50
+𝑘
+50
+k=50
+maximum trajectories. LATS requires every component and operation for optimal performance.
+We also conduct additional experiments on HotPotQA to demonstrate the effect of each component of LATS. We also design a version of ToT and RAP with ReAct prompt and can handle external observations. We use HotPotQA as our setup incorporates both reasoning (through thoughts) and acting (through API calls); the results are shown in Tab.
+5
+. More ablations for token consumption on HotPotQA are in Tab.
+7
+in Appendix
+C
+. Note that baselines generally perform worse than the reasoning-only setting of HotPotQA, which indicates that the acting-based setting is more challenging and adaption of search algorithms to decision-making scenarios is non-trivial.
+Self-reflection.
+We use self-reflection to provide additional semantic signals for the agent. We observe a
+0.05
+0.05
+0.05
+performance drop when removed from LATS, suggesting this is useful. This is a smaller gain Reflexion
+(Shinn et al.,
+2023
+)
+observes over ReAct
+(Yao et al.,
+2023b
+)
+as shown in Tab.
+2
+, suggesting overlap between the types of questions where there is an improvement with self-reflection and search. This variant outperforms RAP-ReAct, reflecting our improvements to MCTS.
+Search Algorithm.
+MCTS is a more principled search algorithm than variants like A* or DFS search and the basis for observed performance gains. We observe the effects of using DFS, and incorporate the LM-based heuristic used in ToT
+(Yao et al.,
+2023a
+)
+in which branches with low values are pruned. This removes the selection and backpropagation operations, and we observe a
+0.08
+0.08
+0.08
+drop in performance when sampling the same number of nodes, but outperforms ToT-ReAct.
+6
+Conclusion
+In this work, we introduce Language Agent Tree Search (LATS), the first framework to unify planning, acting, and reasoning for enhanced LLM problem solving. By deliberately constructing trajectories with search algorithms, incorporating external feedback, and enabling agents to learn from experience, LATS addresses key limitations of prior prompting techniques. Our evaluations demonstrate the ability of LATS to harness LLM capabilities for a variety of decision-making tasks while keeping its reasoning ability without additional training. The proposed synergies between search, interaction, and reflection offer a versatile approach to autonomous decision-making, highlighting the potential of LLMs as generalist agents. A full discussion of the limitations and broader impacts is in Appendix
+B
+.
+References
+Ahn et al. (2022)
+Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng.
+Do as i can, not as i say: Grounding language in robotic affordances.
+arXiv:2204.01691
+, 2022.
+Anthony et al. (2017)
+T. Anthony, Z. Tian, and D. Barber.
+Thinking fast and slow with deep learning and tree search.
+In
+NIPS
+, 2017.
+Austin et al. (2021)
+Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.
+Program synthesis with large language models.
+arXiv:2108.07732
+, 2021.
+Baker et al. (2022)
+Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune.
+Video pretraining (vpt): Learning to act by watching unlabeled online videos.
+arXiv:2206.11795
+, 2022.
+Besta et al. (2023)
+Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler.
+Graph of thoughts: Solving elaborate problems with large language models.
+arXiv:2308.09687
+, 2023.
+Bowman et al. (2015)
+Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning.
+A large annotated corpus for learning natural language inference.
+In
+EMNLP
+, 2015.
+Brown et al. (2020)
+Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
+Language models are few-shot learners.
+In
+NeurIPS
+, 2020.
+Campbell et al. (2002)
+Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu.
+Deep blue.
+Artificial intelligence
+, 2002.
+Chen et al. (2022a)
+Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen.
+Codet: Code generation with generated tests.
+arXiv:2207.10397
+, 2022a.
+Chen et al. (2021)
+Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.
+Evaluating large language models trained on code.
+arXiv:2107.03374
+, 2021.
+Chen et al. (2022b)
+Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen.
+Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.
+arXiv preprint arXiv:2211.12588
+, 2022b.
+Chowdhery et al. (2022)
+Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.
+Palm: Scaling language modeling with pathways.
+arXiv:2204.02311
+, 2022.
+Cobbe et al. (2021)
+Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
+Training verifiers to solve math word problems.
+arXiv:2110.14168
+, 2021.
+Deng et al. (2023)
+Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su.
+Mind2web: Towards a generalist agent for the web.
+arXiv:2306.06070
+, 2023.
+Driess et al. (2023)
+Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.
+Palm-e: An embodied multimodal language model.
+arXiv:2303.03378
+, 2023.
+Du et al. (2023)
+Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel.
+Learning universal policies via text-guided video generation.
+arXiv:2302.00111
+, 2023.
+Evans (2010)
+Jonathan St BT Evans.
+Intuition and reasoning: A dual-process perspective.
+Psychological Inquiry
+, 2010.
+Fan et al. (2022)
+Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar.
+Minedojo: Building open-ended embodied agents with internet-scale knowledge.
+In
+NeurIPS Datasets and Benchmarks Track
+, 2022.
+Furuta et al. (2023)
+Hiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur.
+Multimodal web navigation with instruction-finetuned foundation models.
+arXiv preprint arXiv:2305.11854
+, 2023.
+Gao et al. (2022)
+Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.
+Pal: Program-aided language models.
+arXiv preprint arXiv:2211.10435
+, 2022.
+Guo et al. (2018)
+Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang.
+Long text generation via adversarial training with leaked information.
+AAAI
+, 2018.
+Guss et al. (2019)
+William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov.
+Minerl: A large-scale dataset of minecraft demonstrations.
+In
+IJCAI
+, 2019.
+Hafner et al. (2019)
+Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.
+Learning latent dynamics for planning from pixels.
+In
+ICML
+, 2019.
+Hafner et al. (2023)
+Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.
+Mastering diverse domains through world models.
+arXiv:2301.04104
+, 2023.
+Hao et al. (2023)
+Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu.
+Reasoning with language model is planning with world model.
+arXiv:2305.14992
+, 2023.
+Huang et al. (2023)
+Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou.
+Large language models cannot self-correct reasoning yet.
+arXiv:2310.01798
+, 2023.
+Huang et al. (2022)
+Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al.
+Inner monologue: Embodied reasoning through planning with language models.
+arXiv:2207.05608
+, 2022.
+Jiang et al. (2018)
+D. Jiang, E. Ekwedike, and H. Liu.
+Feedback-based tree search for reinforcement learning.
+In
+ICML
+, 2018.
+Kocsis & Szepesvári (2006)
+Levente Kocsis and Csaba Szepesvári.
+Bandit based monte-carlo planning.
+In
+ECML
+, 2006.
+Kojima et al. (2022)
+Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
+Large language models are zero-shot reasoners.
+arXiv:2205.11916
+, 2022.
+LaValle et al. (2001)
+Steven M LaValle, James J Kuffner, BR Donald, et al.
+Rapidly-exploring random trees: Progress and prospects.
+Algorithmic and computational robotics: new directions
+, 2001.
+Liu et al. (2018)
+Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang.
+Reinforcement learning on web interfaces using workflow-guided exploration.
+In
+ICLR
+, 2018.
+Liu et al. (2023)
+Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang.
+Agentbench: Evaluating llms as agents.
+arXiv:2308.03688
+, 2023.
+Madaan et al. (2023)
+Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.
+Self-refine: Iterative refinement with self-feedback.
+arXiv:2303.17651
+, 2023.
+Nallapati et al. (2016)
+Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang.
+Abstractive text summarization using sequence-to-sequence rnns and beyond.
+In
+SIGNLL
+, 2016.
+Nye et al. (2021)
+Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al.
+Show your work: Scratchpads for intermediate computation with language models.
+arXiv:2112.00114
+, 2021.
+OpenAI (2023)
+OpenAI.
+Gpt-4 technical report.
+arXiv:2303.08774
+, 2023.
+Saparov & He (2022)
+Abulhair Saparov and He He.
+Language models are greedy reasoners: A systematic formal analysis of chain-of-thought.
+arXiv:2210.01240
+, 2022.
+Schick et al. (2023)
+Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
+Toolformer: Language models can teach themselves to use tools.
+arXiv:2302.04761
+, 2023.
+Shen et al. (2023)
+Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.
+Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.
+arXiv:2303.17580
+, 2023.
+Shinn et al. (2023)
+Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
+Reflexion: Language agents with verbal reinforcement learning.
+arXiv:2303.11366
+, 2023.
+Shridhar et al. (2020)
+Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht.
+Alfworld: Aligning text and embodied environments for interactive learning.
+arXiv:2010.03768
+, 2020.
+Silver et al. (2016)
+David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.
+Mastering the game of go with deep neural networks and tree search.
+nature
+, 2016.
+Silver et al. (2017)
+David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, L. Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis.
+Mastering the game of go without human knowledge.
+Nature
+, 2017.
+Sloman (1996)
+Steven A. Sloman.
+The empirical case for two systems of reasoning.
+Psychological Bulletin
+, 1996.
+Sun et al. (2023)
+Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang.
+Adaplanner: Adaptive planning from feedback with language models.
+arXiv:2305.16653
+, 2023.
+Surís et al. (2023)
+Dídac Surís, Sachit Menon, and Carl Vondrick.
+Vipergpt: Visual inference via python execution for reasoning.
+arXiv preprint arXiv:2303.08128
+, 2023.
+Świechowski et al. (2023)
+Maciej Świechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Mańdziuk.
+Monte carlo tree search: A review of recent modifications and applications.
+Artificial Intelligence Review
+, 2023.
+Touvron et al. (2023)
+Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and
+Thomas Scialom.
+Llama 2: Open foundation and fine-tuned chat models.
+arXiv:2307.09288
+, 2023.
+Vodopivec et al. (2017)
+Tom Vodopivec, Spyridon Samothrakis, and Branko Ster.
+On monte carlo tree search and reinforcement learning.
+Journal of Artificial Intelligence Research
+, 2017.
+Wang et al. (2023)
+Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
+Voyager: An open-ended embodied agent with large language models.
+arXiv:2305.16291
+, 2023.
+Wang et al. (2022)
+Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.
+Self-consistency improves chain of thought reasoning in language models.
+arXiv:2203.11171
+, 2022.
+Wei et al. (2022)
+Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.
+Chain of thought prompting elicits reasoning in large language models.
+arXiv:2201.11903
+, 2022.
+Wooldridge & Jennings (1995)
+Michael Wooldridge and Nicholas R Jennings.
+Intelligent agents: Theory and practice.
+The knowledge engineering review
+, 1995.
+Wu et al. (2023)
+Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg.
+Daydreamer: World models for physical robot learning.
+In
+CoRL
+. PMLR, 2023.
+Xie et al. (2023)
+Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie.
+Decomposition enhances reasoning via self-evaluation guided decoding.
+arXiv:2305.00633
+, 2023.
+Yang et al. (2018)
+Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
+Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
+arXiv:1809.09600
+, 2018.
+Yao et al. (2022)
+Shunyu Yao, Howard Chen, John Yang, and Karthik R Narasimhan.
+Webshop: Towards scalable real-world web interaction with grounded language agents.
+In
+NeurIPS
+, 2022.
+Yao et al. (2023a)
+Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.
+Tree of thoughts: Deliberate problem solving with large language models.
+arXiv:2305.10601
+, 2023a.
+Yao et al. (2023b)
+Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
+ReAct: Synergizing reasoning and acting in language models.
+In
+ICLR
+, 2023b.
+Yao et al. (2023c)
+Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese.
+Retroformer: Retrospective large language agents with policy gradient optimization.
+arXiv preprint arXiv:2308.02151
+, 2023c.
+Ye et al. (2021)
+Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao.
+Mastering atari games with limited data.
+In
+NeurIPS
+, 2021.
+Zhou et al. (2022)
+Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi.
+Least-to-most prompting enables complex reasoning in large language models.
+arXiv:2205.10625
+, 2022.
+Zhu et al. (2023)
+Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai.
+Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory.
+arXiv:2305.17144
+, 2023.
+7
+Appendix
+The appendix is organized as follows. First in Sec.
+A
+, we show the pseudocode of our proposed algorithm, LATS; then in Sec.
+B
+, we provide further discussion of our method and its limitations, future direction and broader impact; then in Sec.
+C
+we provide additional experimental results; then in Sec.
+D
+, we specify the environment details in our experiments; finally, we list our prompts used for the three environments in Sec.
+E
+(HotPotQA), Sec.
+F
+(Programming) and Sec.
+G
+(Webshop) respectively.
+Appendix A
+LATS Pseudocode
+Alg.
+1
+shows the pseudocode of our algorithm LATS. Nodes are stored explicitly in the memory. Unless otherwise specified, in all experiments we use
+n
+=
+5
+𝑛
+5
+n=5
+and
+w
+=
+1
+𝑤
+1
+w=1
+.
+Algorithm 1
+LATS
+⁡
+(
+S
+0
+,
+p
+θ
+,
+p
+V
+,
+p
+ref
+,
+d
+,
+k
+,
+n
+,
+w
+)
+LATS
+subscript
+𝑆
+0
+subscript
+𝑝
+𝜃
+subscript
+𝑝
+𝑉
+subscript
+𝑝
+ref
+𝑑
+𝑘
+𝑛
+𝑤
+\operatorname{LATS}(S_{0},p_{\theta},{p_{V}},p_{\text{ref}},d,k,n,w)
+Initial state
+s
+1
+subscript
+𝑠
+1
+s_{1}
+, action generator
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+, value function
+p
+V
+subscript
+𝑝
+𝑉
+p_{V}
+, reflection generator
+p
+ref
+subscript
+𝑝
+ref
+p_{\text{ref}}
+, number of generated actions
+n
+𝑛
+n
+, depth limit
+L
+𝐿
+L
+, number of roll-outs
+K
+𝐾
+K
+, context
+c
+𝑐
+c
+, and exploration weight
+w
+𝑤
+w
+Initialize action space
+A
+𝐴
+A
+, observation space
+O
+𝑂
+O
+Initialize the state-action value function
+p
+V
+:
+S
+×
+A
+↦
+ℝ
+:
+subscript
+𝑝
+𝑉
+maps-to
+𝑆
+𝐴
+ℝ
+{p_{V}}:S\times A\mapsto\mathbb{R}
+and visit counter
+N
+:
+S
+↦
+ℕ
+:
+𝑁
+maps-to
+𝑆
+ℕ
+{N}:S\mapsto\mathbb{N}
+to zero
+for
+k
+←
+0
+,
+…
+,
+K
+−
+1
+←
+𝑘
+0
+…
+𝐾
+1
+k\leftarrow 0,\dots,K-1
+do
+for
+t
+←
+0
+,
+…
+,
+L
+−
+1
+←
+𝑡
+0
+…
+𝐿
+1
+t\leftarrow 0,\dots,L-1
+do
+if
+s
+t
+subscript
+𝑠
+𝑡
+s_{t}
+not terminal
+then
+▷
+▷
+\triangleright
+Expansion & Simulation
+for
+i
+←
+1
+,
+…
+,
+n
+←
+𝑖
+1
+…
+𝑛
+i\leftarrow 1,\dots,n
+do
+Sample
+a
+t
+(
+i
+)
+∼
+p
+θ
+​
+(
+a
+∣
+s
+t
+)
+similar-to
+superscript
+subscript
+𝑎
+𝑡
+𝑖
+subscript
+𝑝
+𝜃
+conditional
+𝑎
+subscript
+𝑠
+𝑡
+a_{t}^{(i)}\sim p_{\theta}(a\mid s_{t})
+Get
+o
+t
+(
+i
+)
+superscript
+subscript
+𝑜
+𝑡
+𝑖
+o_{t}^{(i)}
+from environment,
+s
+t
++
+1
+(
+i
+)
+←
+(
+c
+t
+(
+i
+)
+,
+o
+t
+(
+i
+)
+,
+a
+t
+(
+i
+)
+)
+←
+superscript
+subscript
+𝑠
+𝑡
+1
+𝑖
+superscript
+subscript
+𝑐
+𝑡
+𝑖
+superscript
+subscript
+𝑜
+𝑡
+𝑖
+superscript
+subscript
+𝑎
+𝑡
+𝑖
+s_{t+1}^{(i)}\leftarrow(c_{t}^{(i)},o_{t}^{(i)},a_{t}^{(i)})
+,
+c
+t
++
+1
+(
+i
+)
+←
+(
+o
+t
+(
+i
+)
+,
+a
+t
+(
+i
+)
+)
+←
+superscript
+subscript
+𝑐
+𝑡
+1
+𝑖
+superscript
+subscript
+𝑜
+𝑡
+𝑖
+superscript
+subscript
+𝑎
+𝑡
+𝑖
+c_{t+1}^{(i)}\leftarrow(o_{t}^{(i)},a_{t}^{(i)})
+Evaluate
+V
+t
+(
+i
+)
+∼
+p
+V
+​
+(
+s
+t
+(
+i
+)
+)
+similar-to
+superscript
+subscript
+𝑉
+𝑡
+𝑖
+subscript
+𝑝
+𝑉
+superscript
+subscript
+𝑠
+𝑡
+𝑖
+{V}_{t}^{(i)}\sim{p_{V}}(s_{t}^{(i)})
+▷
+▷
+\triangleright
+Evaluation
+V
+​
+(
+s
+t
+)
+←
+V
+t
+(
+i
+)
+←
+𝑉
+subscript
+𝑠
+𝑡
+superscript
+subscript
+𝑉
+𝑡
+𝑖
+{V}(s_{t})\leftarrow{V}_{t}^{(i)}
+Add
+s
+t
+(
+i
+)
+superscript
+subscript
+𝑠
+𝑡
+𝑖
+s_{t}^{(i)}
+to children
+end
+for
+end
+if
+if
+s
+t
+subscript
+𝑠
+𝑡
+s_{t}
+is terminal
+then
+▷
+▷
+\triangleright
+Reflection
+Get
+r
+𝑟
+r
+from environment
+if
+r
+𝑟
+r
+not success
+then
+reflection
+←
+p
+ref
+​
+(
+c
+t
+)
+←
+reflection
+subscript
+𝑝
+ref
+subscript
+𝑐
+𝑡
+\text{reflection}\leftarrow p_{\text{ref}}(c_{t})
+c
+←
+reflection
+←
+𝑐
+reflection
+c\leftarrow\text{reflection}
+end
+if
+end
+if
+a
+t
+←
+arg
+⁡
+max
+a
+∈
+e
+​
+(
+s
+t
+)
+⁡
+[
+V
+​
+(
+s
+t
+)
++
+w
+​
+ln
+⁡
+N
+​
+(
+s
+t
+−
+1
+)
+N
+​
+(
+s
+t
+)
+]
+←
+subscript
+𝑎
+𝑡
+subscript
+𝑎
+𝑒
+subscript
+𝑠
+𝑡
+𝑉
+subscript
+𝑠
+𝑡
+𝑤
+𝑁
+subscript
+𝑠
+𝑡
+1
+𝑁
+subscript
+𝑠
+𝑡
+a_{t}\leftarrow\arg\max_{a\in e(s_{t})}\left[{V(s_{t})}+w\sqrt{\frac{\ln{N}(s_{t-1})}{{N}(s_{t})}}\right]
+▷
+▷
+\triangleright
+Selection
+N
+​
+(
+s
+t
++
+1
+)
+←
+N
+​
+(
+s
+t
++
+1
+)
++
+1
+←
+𝑁
+subscript
+𝑠
+𝑡
+1
+𝑁
+subscript
+𝑠
+𝑡
+1
+1
+{N}(s_{t+1})\leftarrow{N}(s_{t+1})+1
+if
+a
+t
+subscript
+𝑎
+𝑡
+a_{t}
+is an output action
+then
+break
+end
+for
+T
+←
+←
+𝑇
+absent
+T\leftarrow
+the actual number of steps
+for
+t
+←
+T
+−
+1
+,
+…
+,
+0
+←
+𝑡
+𝑇
+1
+…
+0
+t\leftarrow T-1,\dots,0
+do
+▷
+▷
+\triangleright
+Backpropagation
+V
+​
+(
+s
+t
+)
+←
+V
+​
+(
+s
+t
+)
+​
+(
+N
+​
+(
+s
+t
+)
+−
+1
+)
++
+r
+N
+​
+(
+s
+t
+)
+←
+𝑉
+subscript
+𝑠
+𝑡
+𝑉
+subscript
+𝑠
+𝑡
+𝑁
+subscript
+𝑠
+𝑡
+1
+𝑟
+𝑁
+subscript
+𝑠
+𝑡
+V(s_{t})\leftarrow\frac{V(s_{t})(N(s_{t})-1)+r}{N(s_{t})}
+end
+for
+end
+for
+Appendix B
+Discussion
+Limitations.
+Although LATS can improve reasoning and decision-making, this arrives at a higher computational cost relative to simpler prompting methods like ReAct or Reflexion. The search process takes more time than standard prompting or simpler techniques, and requires greater inference costs. While such an issue is mitigated by the fact that the number of nodes
+n
+𝑛
+n
+expanded at every step provides a natural trade-off between performance and efficiency (setting
+n
+=
+1
+𝑛
+1
+n=1
+makes the method as effecient as ReAct with multiple trials or CoT-SC), in practice we recommend using LATS for difficult tasks like programming or for situations where performance is prioritized over efficiency. We hope that continued advancements in LLMs will reduce costs and increase the practicality of LATS.
+Additionally, the benchmarks we use in this paper are relatively simple and focused on decision-making, compared to the complexity of real-world interactive environments. In addition, some environments might not easily support rollbacks to previous states. However, the design of LATS is flexible and can be adjusted to various resource constraints. Using planning-based prompting methods like LATS in environments like Minecraft
+(Fan et al.,
+2022
+)
+and more reasoning benchmarks would be interesting avenues for future work.
+Broader impact.
+LATS is a framework that enhances LLM performance through interactions with an environment. This improvement in autonomous decision-making may facilitate harmful uses of LLMs. Alternatively, LATS enhances interpretability and the potential for greater alignment, as it generates understandable, high-level linguistic reasoning and actions through several rounds of decision-making and reflection, rather than relying on implicit, low-level token values.
+Appendix C
+Ablations
+Prompt Method
+HotpotQA (EM)
+LATS (w=0.5)
+0.55
+LATS (w=2.0)
+0.61
+LATS (d=4)
+0.58
+LATS (CoT)
+0.60
+LATS (No LM Heuristic)
+0.37
+LATS
+0.61
+Table 6:
+Ablation results on LATS and baseline variants in HotPotQA measured by Exact Match (EM). We test different depth
+d
+𝑑
+d
+, exploration factor
+w
+𝑤
+w
+, and versions of LATS using CoT and without the LM value function. We sample
+n
+=
+5
+𝑛
+5
+n=5
+and
+k
+=
+50
+𝑘
+50
+k=50
+trajectories.
+Figure 4:
+Performance over successive iterations on HumanEval with GPT-3.5.
+In this section, we ablate various designs of LATS. Experiments are conducted on HotPotQA with a maximum of
+k
+=
+50
+𝑘
+50
+k=50
+trajectories and sampling size of
+n
+=
+5
+𝑛
+5
+n=5
+and HumanEval with a maximum of
+k
+=
+8
+𝑘
+8
+k=8
+trajectories and sampling size of
+n
+=
+5
+𝑛
+5
+n=5
+. The result for HotPotQA is shown in Tab.
+5
+and HumanEval in Fig.
+4
+.
+Exploration weight.
+We find that there is lower performance on HotPotQA when the exploration weight
+w
+𝑤
+w
+in the selection formula is decreased to
+0.5
+0.5
+0.5
+, suggesting that this reduces the effectiveness of the search. Increasing
+w
+𝑤
+w
+to
+2.0
+2.0
+2.0
+does not lead to a performance improvement, but we tend to observe faster convergence. The optimal setting depends on the particular environment and complexity of the state space.
+Depth.
+In our main experiments we use a maximum depth of
+d
+=
+7
+𝑑
+7
+d=7
+on HotPotQA for all methods, following previous work
+(Yao et al.,
+2023b
+)
+. We ablate the effect on LATS after reducing it to
+d
+=
+4
+𝑑
+4
+d=4
+. This results in only a slight drop in performance. We find that most questions can be answered within four steps, and using a greater number of steps tends to force the agent into local minima and rarely improves success.
+LM value function.
+The LM value function scores states based on expected future reward. Without this heuristic, the only signal to guide search would be from environment rewards for completed trajectories, which are scarce and often binary. When we remove the evaluation operation, we observe a dramatic
+0.24
+0.24
+0.24
+drop in performance.
+Performance over time.
+To see the effects of increasing the number of trajectories sampled, we change
+k
+𝑘
+k
+to different values. We conduct this experiment on HumanEval, which has a more noticeable difference due to sampling less trajectories. The results are shown in Fig.
+4
+, in which LATS scales better with more iterations than Reflexion.
+Sample complexity and Token cost.
+One possible concern of LATS is that the tree-structured search might consume much more tokens than existing methods. To further study the computational cost of LATS compared to prior methods, we examine the sample complexity (i.e. asymptotic token cost) of all methods considered in this paper, and count the average number of nodes expanded by our method and other tree-structured methods (ToT and RAP) upon successful search on HotPotQA. We present the results in Tab.
+7
+; the result shows that our method has the same sample complexity as other tree-based search methods, and has less average number of nodes expanded upon success, which indicates less token cost. The token cost gap will be even larger when taking failed trajectories into account, since our method has higher success rate and reaches computational budget limit less often.
+Method
+Performance (
+↑
+↑
+\uparrow
+)
+Sample complexity (
+↓
+↓
+\downarrow
+)
+Avg. #nodes upon success (
+↓
+↓
+\downarrow
+)
+ReAct (Best
+k
+=
+250
+𝑘
+250
+k=250
+)
+0.42
+0.42
+0.42
+O
+​
+(
+k
+)
+𝑂
+𝑘
+O(k)
+N/A
+CoT-SC (
+n
+=
+1
+,
+k
+=
+250
+formulae-sequence
+𝑛
+1
+𝑘
+250
+n=1,k=250
+)
+0.40
+0.40
+0.40
+O
+​
+(
+k
+)
+𝑂
+𝑘
+O(k)
+N/A
+LATS (
+n
+=
+1
+,
+k
+=
+50
+formulae-sequence
+𝑛
+1
+𝑘
+50
+n=1,k=50
+)
+0.48
+0.48
+0.48
+O
+​
+(
+k
+)
+𝑂
+𝑘
+O(k)
+N/A
+ToT (ReAct)
+0.49
+0.49
+0.49
+O
+​
+(
+k
+​
+n
+)
+𝑂
+𝑘
+𝑛
+O(kn)
+84.05
+84.05
+84.05
+RAP (ReAct)
+0.54
+0.54
+0.54
+O
+​
+(
+k
+​
+n
+)
+𝑂
+𝑘
+𝑛
+O(kn)
+70.60
+70.60
+70.60
+LATS (
+n
+=
+5
+,
+k
+=
+50
+formulae-sequence
+𝑛
+5
+𝑘
+50
+n=5,k=50
+)
+0.61
+0.61
+0.61
+O
+​
+(
+k
+​
+n
+)
+𝑂
+𝑘
+𝑛
+O(kn)
+66.65
+66.65
+66.65
+Table 7:
+The performance, sample complexity of different methods and average number of nodes expanded upon success by methods with tree-based search.
+n
+𝑛
+n
+is the number of children nodes expanded at every step and
+k
+𝑘
+k
+is the number of trajectories. Our method has the same sample complexity as other methods with tree-based search and expands less nodes upon success, which indicates lower token cost.
+Appendix D
+Environment Details
+D.1
+HotPotQA
+Figure 5:
+Example trajectories on HotPotQA for ReAct (left) and LATS (right). LATS can sample more actions and avoid failure from previous mistakes by evaluating states with an LM to guide the search toward promising areas of the tree.
+HotPotQA
+(Yang et al.,
+2018
+)
+is a question-answering dataset that requires reasoning over multiple supporting documents to answer questions. It contains 113k Wikipedia-based question-answer pairs crafted by crowdworkers to be diverse, multi-hop, and explainable. Questions cover a range of types like entities, locations, dates, and comparison of shared properties between two entities. Crowdworkers also provide supporting facts from the documents that justify the answer. We use the HotPotQA benchmark setting with all the Wikipedia paragraphs to test retrieval. We use a randomly selected subset of 100 questions for our experiments and a maximum depth limit of 6. Fig.
+5
+illustrates how ReAct and LATS work on an example task of HotPotQA, and gives a qualitative example on how LATS outperforms ReAct on the task.
+Action Space.
+We adopt the Wikipedia web API proposed in
+Yao et al. (
+2023b
+)
+, with three types of actions to support interactive information retrieval:
+(1)
+search
+[
+entity
+], which returns the first 5 sentences from the corresponding
+entity
+wiki page if it exists, or else suggests top-5 similar entities from the Wikipedia search engine,
+(2)
+lookup
+[
+string
+], which returns the next sentence in the page containing
+string
+,
+(3)
+finish
+[
+answer
+], which finishes the current task with
+answer
+.
+These API calls and free-form thoughts form the action space for this environment.
+D.2
+Programming
+The HumanEval dataset
+(Chen et al.,
+2021
+)
+is a collection of 164 handwritten programming problems introduced to evaluate the functional correctness of models for synthesizing programs from natural language descriptions. Each problem includes a function signature, docstring description, reference implementation, and multiple unit tests, with an average of 7.7 tests per problem. The programming tasks assess comprehension of natural language, reasoning, algorithms, and basic mathematics, at a difficulty level comparable to simple software interview questions. Pass rates are evaluated with the pass@k metric, where k samples are generated per problem and a problem is considered solved if any sample passes all tests. We use all 164 problems for our experiments and a maximum depth limit of 8.
+The Mostly Basic Programming Problems (MBPP)
+Austin et al. (
+2021
+)
+benchmark contains 974 short Python functions designed to evaluate program synthesis techniques. The dataset was constructed by crowdsourcing from workers with basic Python knowledge. Each data point consists of a natural language description of a programming task, a reference solution implementation, and three test cases for functional correctness. The natural language prompts are typically short, one-sentence descriptions. Solutions cover common programming constructs including mathematical operations, list processing, string manipulation, and usage of the Python standard library. On average, solutions are 6.8 lines of code. The dataset is also supplemented with an additional set of 426 problems that were manually verified for unambiguous specifications, standard function signatures, and accurate test cases. We use a randomly selected subset of 397 problems for our experiments.
+D.3
+WebShop
+WebShop
+(Yao et al.,
+2022
+)
+is an interactive web-based environment designed to evaluate agents on grounded language understanding and decision-making. It simulates an e-commerce shopping task by providing agents with over 1 million real-world products scraped from Amazon, spanning 5 categories and 113 subcategories. These products contain rich linguistic information, with an average text length of 262 words and a vocabulary size of 224k. In addition, there are over 800k unique product options available for customization. The environment renders webpages in two modes: HTML mode provides pixel-level observations with interactive elements, while simple mode converts the raw HTML into a structured text observation more amenable for training agents. The action space consists of query searches and button clicks, which transition between 4 page types: search, results, item and item-detail. Instructions are crowdsourced natural language specifying product attributes and options, with a total of 12k collected. Automatic rewards are computed by comparing the product purchased by the agent against the attributes and options specified in the instruction, using both lexical matching and semantic similarity metrics.
+Type
+Argument
+State
+→
+→
+\rightarrow
+Next State
+search
+[
+Query
+]
+Search
+→
+→
+\rightarrow
+Results
+choose
+Back to search
+∗
+*
+→
+→
+\rightarrow
+Search
+choose
+Prev/Next page
+Results
+→
+→
+\rightarrow
+Results
+choose
+[
+Product title
+]
+Results
+→
+→
+\rightarrow
+Item
+choose
+[
+Option
+]
+Item
+→
+→
+\rightarrow
+Item
+choose
+Desc/Overview
+Item
+→
+→
+\rightarrow
+Item-Detail
+choose
+Previous
+Item-Detail
+→
+→
+\rightarrow
+Item
+choose
+Buy
+Item
+→
+→
+\rightarrow
+Episode End
+Table 8:
+Action space of webshop.
+There are two evaluation metrics used in WebShop: (1)
+Task Score
+: defined as
+(
+100
+×
+avg. reward
+)
+100
+avg. reward
+(100\times\text{avg. reward})
+, which captures the average reward obtained across episodes; and (2)
+Success Rate (SR)
+defined as the portion of instructions where
+r
+=
+1
+𝑟
+1
+r=1
+. The reward is calculated based on the number of attributes satisfied by the selected item. We use 50 environments for our experiments and a maximum depth limit of 15.
+Appendix E
+HotPotQA Prompts
+E.1
+Base Acting Prompt
+Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types:
+(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
+(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
+(3) Finish[answer], which returns the answer and finishes the task.
+After each observation, provide the next Thought and next Action.
+Here are some examples:
+Question: Which magazine was started first Arthur’s Magazine or First for Women?
+Thought 1: I need to search Arthur’s Magazine and First for Women, and find which was started first.
+Action 1: Search[Arthur’s Magazine]
+Observation 1: Arthur’s Magazine (18440̆0e20̆0800̆0931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey’s Lady’s Book.[3]
+Thought 2: Arthur’s Magazine was started in 1844. I need to search First for Women next.
+Action 2: Search[First for Women]
+Observation 2: First for Women is a woman’s magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey.[2] In 2011 the circulation of the magazine was 1,310,696 copies.[3]
+Thought 3: First for Women was started in 1989. 1844 (Arthur’s Magazine) ¡ 1989 (First for Women), so Arthur’s Magazine was started first.
+Action 3: Finish[Arthur’s Magazine]
+(examples)
+You have attempted to answer the following question before and failed. The following reflection(s) give a plan to avoid failing to answer the question in the same way you did previously. Use them to improve your strategy of correctly answering the given question.
+(trajectories)
+(input)
+E.2
+Base Reasoning Prompt
+Solve a question answering task by having a Thought, then Finish with your answer. Thought can reason about the current situation. Finish[answer] returns the answer and finishes the task. You will be given context that you should use to help you answer the question. Start your response with either Action or an indexed Thought
+Here are some examples:
+Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?
+Let’s think step by step.
+Thought 1: The eastern sector of Colorado orogeny extends into the High Plains.
+Thought 2: High Plains rise in elevation from around 1,800 to 7,000 ft
+Thought 3: The answer is 1,800 to 7,000 ft.
+Action: Finish[1,800 to 7,000 ft]
+(examples)
+Previous trial:
+(trajectories)
+(input)
+E.3
+Value Function Prompt
+Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by environmental observations about the situation, thoughts that can reason about the current situation and actions that can be three types:
+(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
+(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
+(3) Finish[answer], which returns the answer and finishes the task.
+Given a question and a trajectory, evaluate its correctness and provide your reasoning and analysis in detail. Focus on the latest thought, action, and observation. Incomplete trajectories can be correct if the thoughts and actions so far are correct, even if the answer is not found yet. Do not generate additional thoughts or actions. Then at the last line conclude ”Thus the correctness score is s”, where s is an integer from 1 to 10.
+Question: Which magazine was started first Arthur’s Magazine or First for Women?
+Thought 1: I need to search Arthur’s Magazine and First for Women, and find which was started first.
+Action 1: Search[Arthur’s Magazine]
+Observation 1: Arthur’s Magazine (18440̆0e20̆0800̆0931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey’s Lady’s Book.[3]
+This trajectory is correct as it is reasonable to search for the first magazine provided in the question. It is also better to have simple searches corresponding to a single entity, making this the best action.
+Thus the correctness score is 10
+(other examples)
+(failed trajectories)
+(context)
+E.4
+Reflection Prompt
+Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by environmental observations about the situation, thoughts that can reason about the current situation and actions that can be three types:
+(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
+(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
+(3) Finish[answer], which returns the answer and finishes the task.
+Given a question and a trajectory, evaluate its correctness and provide your reasoning and analysis in detail. Focus on the latest thought, action, and observation. Incomplete trajectories can be correct if the thoughts and actions so far are correct, even if the answer is not found yet. Do not generate additional thoughts or actions. Then at the last line conclude ”Thus the correctness score is s”, where s is an integer from 1 to 10.
+Question: Which magazine was started first Arthur’s Magazine or First for Women?
+Thought 1: I need to search Arthur’s Magazine and First for Women, and find which was started first.
+Action 1: Search[Arthur’s Magazine]
+Observation 1: Arthur’s Magazine (18440̆0e20̆0800̆0931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey’s Lady’s Book.[3]
+This trajectory is correct as it is reasonable to search for the first magazine provided in the question. It is also better to have simple searches corresponding to a single entity, making this the best action.
+Thus the correctness score is 10
+(other examples)
+(failed trajectories)
+(context)
+Appendix F
+Programming Prompts
+F.1
+HumanEval function implementation example
+Sample function signature:
+⬇
+def
+minSubArraySum
+(
+nums
+):
+Given
+an
+array
+of
+integers
+nums
+,
+find
+the
+minimum
+sum
+of
+any
+non
+-
+empty
+sub
+-
+array
+of
+nums
+.
+Example
+minSubArraySum
+([2,
+3,
+4,
+1,
+2,
+4])
+==
+1
+minSubArraySum
+([-1,
+-2,
+-3])
+==
+-6
+Sample function body implementation:
+⬇
+min_sum
+=
+float
+(’
+inf
+’)
+for
+i
+in
+range
+(
+len
+(
+nums
+)):
+current_sum
+=
+0
+for
+j
+in
+range
+(
+i
+,
+len
+(
+nums
+)):
+current_sum
++=
+nums
+[
+j
+]
+if
+current_sum
+<
+min_sum
+:
+min_sum
+=
+current_sum
+return
+min_sum
+F.2
+Base Acting/Reasoning Prompt
+You are an AI Python assistant. You will be given your previous implementation of a function, a series of unit tests results, and your self-reflection on your previous implementation. Write your full implementation (restate the function signature).
+Example 1:
+[previous impl]:
+⬇
+def
+add
+(
+a
+:
+int
+,
+b
+:
+int
+)
+->
+int
+:
+”””
+Given
+integers
+a
+and
+b
+,
+return
+the
+total
+value
+of
+a
+and
+b
+.
+”””
+return
+a
+-
+b
+[unit test results from previous impl]:
+Tested passed:
+Tests failed:
+assert add(1, 2) == 3 # output: -1
+assert add(1, 2) == 4 # output: -1
+[reflection on previous impl]:
+The implementation failed the test cases where the input integers are 1 and 2. The issue arises because the code does not add the two integers together, but instead subtracts the second integer from the first. To fix this issue, we should change the operator from ‘-‘ to ‘+‘ in the return statement. This will ensure that the function returns the correct output for the given input.
+[improved impl]:
+⬇
+def
+add
+(
+a
+:
+int
+,
+b
+:
+int
+)
+->
+int
+:
+”””
+Given
+integers
+a
+and
+b
+,
+return
+the
+total
+value
+of
+a
+and
+b
+.
+”””
+return
+a
++
+b
+F.3
+Reflection Prompt
+You are a Python programming assistant. You will be given a function implementation and a series of unit test results. Your goal is to write a few sentences to explain why your implementation is wrong as indicated by the tests. You will need this as guidance when you try again later. Only provide the few sentence description in your answer, not the implementation. You will be given a few examples by the user.
+Example 1:
+[previous impl]:
+⬇
+def
+add
+(
+a
+:
+int
+,
+b
+:
+int
+)
+->
+int
+:
+”””
+Given
+integers
+a
+and
+b
+,
+return
+the
+total
+value
+of
+a
+and
+b
+.
+”””
+return
+a
+-
+b
+[unit test results from previous impl]:
+Tested passed:
+Tests failed:
+assert add(1, 2) == 3 # output: -1
+assert add(1, 2) == 4 # output: -1
+[reflection on previous impl]:
+The implementation failed the test cases where the input integers are 1 and 2. The issue arises because the code does not add the two integers together, but instead subtracts the second integer from the first. To fix this issue, we should change the operator from ‘-‘ to ‘+‘ in the return statement. This will ensure that the function returns the correct output for the given input.
+F.4
+Test Case Generation Prompt
+You are an AI coding assistant that can write unique, diverse, and intuitive unit tests for functions given the signature and docstring.
+(input)
+Appendix G
+WebShop Prompts
+G.1
+Acting Prompt
+Webshop
+Instruction:
+i would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars
+[Search]
+Action: search[3 ounce bright citrus deodorant sensitive skin]
+Observation:
+[Back to Search]
+Page 1 (Total results: 50)
+[Next
+]
+[B078GWRC1J]
+Bright Citrus Deodorant by Earth Mama — Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
+$10.99
+[B078GTKVXY]
+Ginger Fresh Deodorant by Earth Mama — Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
+$10.99
+[B08KBVJ4XN]
+Barrel and Oak - Aluminum-Free Deodorant, Deodorant for Men, Essential Oil-Based Scent, 24-Hour Odor Protection, Cedar & Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
+$15.95
+Action: think[B078GWRC1J and B078GTKVXY are bright citrus deodorant less then 50 dollars. I can check B078GWRC1J first.]
+Observation: OK.
+Action: click[B078GWRC1J]
+Observation:
+[Back to Search]
+[Prev]
+scent [assorted scents][bright citrus][calming lavender][ginger fresh][simply non-scents]
+size [travel set (4-pack)][3 ounce (pack of 1)][3-ounce (2-pack)]
+Bright Citrus Deodorant by Earth Mama — Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
+Price: $10.99
+Rating: N.A.
+[Description]
+[Features]
+[Reviews]
+[Buy Now]
+Action: think[For 3 ounce bottle of bright citrus deodorant for sensitive skin, the item has options ’bright citrus’ and ’3 ounce (pack of 1)’ and seems good to buy.]
+Observation: OK.
+Action: click[bright citrus]
+Observation: You have clicked bright citrus.
+Action: click[3 ounce (pack of 1)]
+Observation: You have clicked 3 ounce (pack of 1).
+Action: click[Buy Now]
+G.2
+Value Function Prompt
+Given an item to purchase and a trajectory that aims to buy an item that exactly matches the specification, which corresponds to the ideal score of 1.0, analyze the following trajectory, then at the last line conclude ”Thus the correctness score is s”, where s is an integer from 1 to 10.
+Here are some examples
+Webshop
+Instruction:
+i am looking for dairy free and apple variety pack of chips, and price lower than 30.00 dollars
+[Search]
+Action: search[dairy free and apple variety pack of chips]
+Observation:
+[Back to Search]
+Page 1 (Total results: 50)
+[Next]
+[B07HRFJWP8]
+Enjoy Life Foods Soft Baked Ovals, Breakfast Bars, Nut Free Bars, Soy Free, Dairy Free, Non GMO, Gluten Free, Vegan, Variety Pack, 4 Boxes (20 Bars Total)
+$100.0
+[B01KMHY5PG]
+Enjoy Life Soft Baked Chewy Bars, Variety Pack, Nut Free Bars, Soy Free, Dairy Free, Gluten Free, 6 Boxes (30 Total Bars)
+$21.49
+[B008D2X8C4]
+Enjoy Life Lentil Chips Variety Pack, Dairy Free Chips, Soy Free, Nut Free, Non GMO, Vegan, Gluten Free, 24 Bags (0.8 oz)
+$100.0
+Action: think[The first two items are not what I want. The third item is the dairy free and apple variety pack of chips I want.]
+Observation: OK.
+Action: click[B008D2X8C4]
+Observation:
+[Back to Search]
+[Prev]
+flavor name [dill and sour cream][garlic & parmesan][light sea salt][margherita pizza][thai chili lime][variety pack]
+size [0.8 ounce (pack of 24)][4 ounce (pack of 12)]
+Price: $100.0
+Rating: N.A.
+[Description]
+[Features]
+[Reviews]
+[Buy Now]
+Action: think[The item has options ’variety pack’ and ’0.8 ounce (pack of 24)’ and seems good to buy.]
+Observation: OK.
+Action: click[variety pack]
+Observation: You have clicked variety pack.
+Action: click[0.8 ounce (pack of 24)]
+Observation: You have clicked 0.8 ounce (pack of 24).
+Action: click[Buy Now]
+Reflection: In this attempt, I was unsuccessful. I accidentally bought a product that was $100, which is more than my budget of $30. Either way, the initial search results were not good. Next time, I will do search[”variety pack of chips”] and then check if the results meet the dairy free and the $30 budget constraints. I will continue to refine my searches so that I can find more products.
+Thus the correctness score is 5
+(input)
+G.3
+Reflection Prompt
+Given an item to purchase and a trajectory that aims to buy an item that exactly matches the specification, which corresponds to the ideal score of 1.0, analyze the following trajectory, then at the last line conclude ”Thus the correctness score is s”, where s is an integer from 1 to 10.
+Here are some examples
+Webshop
+Instruction:
+i am looking for dairy free and apple variety pack of chips, and price lower than 30.00 dollars
+[Search]
+Action: search[dairy free and apple variety pack of chips]
+Observation:
+[Back to Search]
+Page 1 (Total results: 50)
+[Next]
+[B07HRFJWP8]
+Enjoy Life Foods Soft Baked Ovals, Breakfast Bars, Nut Free Bars, Soy Free, Dairy Free, Non GMO, Gluten Free, Vegan, Variety Pack, 4 Boxes (20 Bars Total)
+$100.0
+[B01KMHY5PG]
+Enjoy Life Soft Baked Chewy Bars, Variety Pack, Nut Free Bars, Soy Free, Dairy Free, Gluten Free, 6 Boxes (30 Total Bars)
+$21.49
+[B008D2X8C4]
+Enjoy Life Lentil Chips Variety Pack, Dairy Free Chips, Soy Free, Nut Free, Non GMO, Vegan, Gluten Free, 24 Bags (0.8 oz)
+$100.0
+Action: think[The first two items are not what I want. The third item is the dairy free and apple variety pack of chips I want.]
+Observation: OK.
+Action: click[B008D2X8C4]
+Observation:
+[Back to Search]
+[Prev]
+flavor name [dill and sour cream][garlic & parmesan][light sea salt][margherita pizza][thai chili lime][variety pack]
+size [0.8 ounce (pack of 24)][4 ounce (pack of 12)]
+Price: $100.0
+Rating: N.A.
+[Description]
+[Features]
+[Reviews]
+[Buy Now]
+Action: think[The item has options ’variety pack’ and ’0.8 ounce (pack of 24)’ and seems good to buy.]
+Observation: OK.
+Action: click[variety pack]
+Observation: You have clicked variety pack.
+Action: click[0.8 ounce (pack of 24)]
+Observation: You have clicked 0.8 ounce (pack of 24).
+Action: click[Buy Now]
+Reflection: In this attempt, I was unsuccessful. I accidentally bought a product that was $100, which is more than my budget of $30. Either way, the initial search results were not good. Next time, I will do search[”variety pack of chips”] and then check if the results meet the dairy free and the $30 budget constraints. I will continue to refine my searches so that I can find more products.
+(input)
+Reflection:
+◄
+Feeling
+lucky?
+Conversion
+report
+Report
+an issue
+View original
+on arXiv
+►

research/notes/231004406-language-agent-tree-search-unifies-reasoning-acting-and-planning-in-la-3.md

@@ -0,0 +1,4095 @@
+---
+title: '[2310.04406] Language Agent Tree Search Unifies Reasoning Acting and Planning
+  in Language Models'
+id: 231004406-language-agent-tree-search-unifies-reasoning-acting-and-planning-in-la-3
+tags:
+- deepread
+created: '2026-06-10T00:40:52.405072Z'
+source: https://ar5iv.labs.arxiv.org/html/2310.04406
+source_domain: ar5iv.labs.arxiv.org
+fetched_at: '2026-06-10T00:40:52.404928Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2310.04406] Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
+Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
+Andy Zhou
+University of Illinois at Urbana-Champaign
+AI@UIUC
+Kai Yan
+University of Illinois at Urbana-Champaign
+Michal Shlapentokh-Rothman
+University of Illinois at Urbana-Champaign
+Haohan Wang
+University of Illinois at Urbana-Champaign
+Yu-Xiong Wang
+University of Illinois at Urbana-Champaign
+Abstract
+While large language models (LLMs) have demonstrated impressive performance on a range of decision-making tasks, they rely on simple acting processes and fall short of broad deployment as autonomous agents. We introduce LATS (Language Agent Tree Search), a general framework that synergizes the capabilities of LLMs in planning, acting, and reasoning. Drawing inspiration from Monte Carlo tree search commonly used in model-based reinforcement learning, LATS employs LLMs as agents, value functions, and optimizers, repurposing their latent strengths for enhanced decision-making. What is crucial in this method is the use of an environment for external feedback, which offers a more deliberate and adaptive problem-solving mechanism that moves beyond the limitations of existing techniques. Our experimental evaluation across diverse domains, such as programming, HotPotQA, and WebShop, illustrates the applicability of LATS for decision-making while maintaining competitive reasoning performance. In particular, LATS achieves 94.4% for programming on HumanEval with GPT-4 and an average score of 75.9 for web browsing on WebShop with GPT-3.5, demonstrating the effectiveness and generality of our method.
+1
+Introduction
+General autonomous agents capable of reasoning and decision-making in a variety of environments
+(Wooldridge & Jennings,
+1995
+)
+have been of longstanding interest in the field of artificial intelligence. While this has traditionally been studied in reinforcement learning, the recent rise of large language models (LLMs)
+(Brown et al.,
+2020
+; Chowdhery et al.,
+2022
+; Touvron et al.,
+2023
+; OpenAI,
+2023
+)
+with strong reasoning and general adaptability offers an alternative paradigm. Not only have LLMs excelled on standard NLP tasks such as text summarization
+(Nallapati et al.,
+2016
+)
+or natural language inference
+(Bowman et al.,
+2015
+)
+, but they have been adapted to an increasingly diverse set of tasks that often require advanced common-sense reasoning or quantitative skills
+(Cobbe et al.,
+2021
+; Saparov & He,
+2022
+)
+. LLMs are also capable of performing in complex environments that involve knowledge and reasoning, such as web navigation
+(Yao et al.,
+2022
+; Deng et al.,
+2023
+)
+, tool-use
+(Schick et al.,
+2023
+)
+, or open-ended games
+(Fan et al.,
+2022
+)
+.
+Figure 1:
+An overview of LATS. LATS uses an external environment and self-reflection to improve reasoning and decision-making.
+Reasoning and acting abilities have also been improved by prompting techniques that augment LLMs with feedback or observations from an external environment
+(Yao et al.,
+2023b
+; Gao et al.,
+2022
+; Shinn et al.,
+2023
+)
+. This eliminates the need to rely entirely on the base abilities of the Language Model (LM), enhancing it through external tools or semantic feedback. Despite this strength, these methods are reflexive and fall short of humans’ deliberate and thoughtful decision-making characteristics to solve problems
+(Sloman,
+1996
+; Evans,
+2010
+)
+. In particular, such methods fail to consider multiple reasoning paths or to plan ahead. Recent search-guided LLM works
+(Xie et al.,
+2023
+; Yao et al.,
+2023a
+; Hao et al.,
+2023
+)
+address this issue by searching over multiple reasoning chains. While these methods enable planning, these methods operate in isolation and do not incorporate external feedback that can improve reasoning.
+To help address these issues, we propose LATS (Language Agent Tree Search), a general framework for decision-making and reasoning with language models. LATS unifies LM planning, acting, and reasoning strategies by expanding ReAct
+(Yao et al.,
+2023b
+)
+into a search over a combinatorial space of possible reasoning and acting steps. We adapt Monte Carlo tree search (MCTS) from model-based reinforcement learning
+(Silver et al.,
+2017
+; Anthony et al.,
+2017
+; Jiang et al.,
+2018
+)
+to language agents, repurposing a pretrained LLM as an agent, value function, and optimizer. Utilizing the strong natural language understanding and in-context learning ability of modern LMs, we use text as an interface between each component of the framework, allowing LATS to adapt planning to environmental conditions without additional training. To the best of our knowledge,
+LATS is the first framework that combines reasoning, acting, and planning to enhance LLMs
+. Notably, LATS doubles the performance of GPT-3.5 on HotPotQA
+(Yang et al.,
+2018
+)
+over ReAct
+(Yao et al.,
+2023b
+)
+and raises the average score by
+22.1
+22.1
+22.1
+on WebShop
+(Yao et al.,
+2022
+)
+. When used with GPT-4, LATS achieves a
+94.4
+94.4
+94.4
+Pass@1 rate for programming on HumanEval
+(Chen et al.,
+2021
+)
+, setting the state of the art. To summarize, our
+contributions
+are the following:
+•
+We introduce an LM-based Monte Carlo tree search variant to deliberately construct the best trajectory from sampled actions, enabling more flexible and adaptive problem-solving compared to reflexive prompting methods. This is guided by heuristics from the LM.
+•
+By integrating external feedback and self-reflection, LATS enhances model sensibility and enables agents to learn from experience, surpassing reasoning-based search methods.
+•
+Through experiments across diverse domains like programming, interactive QA, and web navigation, we demonstrate the versatility of LATS in harnessing LLMs for autonomous reasoning and decision-making.
+2
+Related Work
+Approach
+Reasoning
+Acting
+Planning
+Self
+External
+Reflection
+Memory
+CoT
+(Wei et al.,
+2022
+)
+✓
+×
+\times
+×
+\times
+×
+\times
+×
+\times
+ReAct
+(Yao et al.,
+2023b
+)
+✓
+✓
+×
+\times
+×
+\times
+×
+\times
+ToT
+(Yao et al.,
+2023a
+)
+✓
+×
+\times
+✓
+✓
+✓
+RAP
+(Hao et al.,
+2023
+)
+✓
+×
+\times
+✓
+×
+\times
+✓
+Self-Refine
+(Madaan et al.,
+2023
+)
+✓
+×
+\times
+×
+\times
+✓
+×
+\times
+Beam Search
+(Xie et al.,
+2023
+)
+✓
+×
+\times
+×
+\times
+✓
+×
+\times
+Reflexion
+(Shinn et al.,
+2023
+)
+✓
+✓
+×
+\times
+✓
+✓
+LATS (Ours)
+✓
+✓
+✓
+✓
+✓
+Table 1:
+A summary of related work on reasoning, acting, and planning. LATS is the first work incorporating designs from all three domains, allowing use in all corresponding tasks. We refer to planning as the use of a search algorithm, self-reflection as the use of LM-generated feedback, and external memory as storaging past text context for future updates of solution.
+a) Tree-of-Thoughts
+b) Reasoning via Planning
+c) Language Agent Tree Search
+Figure 2:
+An overview of the differences between LATS and recently proposed LM search algorithms ToT
+(Yao et al.,
+2023a
+)
+and RAP
+(Hao et al.,
+2023
+)
+. LATS leverages environmental feedback and self-reflection to further adapt search and improve performance.
+LLMs for reasoning.
+For LLMs, reasoning typically involves decomposing complex inputs into sequential intermediate steps towards a final answer
+(Cobbe et al.,
+2021
+)
+, demonstrated with Chain-of-Thought (CoT) prompting
+(Wei et al.,
+2022
+)
+and its variants
+(Wei et al.,
+2022
+; Kojima et al.,
+2022
+; Wang et al.,
+2022
+)
+. However, these methods, which create chains autoregressively in a single step, often suffer from error propagation as the number of steps increases
+(Guo et al.,
+2018
+; Chen et al.,
+2022b
+)
+due to compound errors. Various advancements aim to mitigate this issue; some approaches, such as Self-Consistency
+(Wang et al.,
+2022
+)
+, employ majority voting over sampled chains, while others focus on multi-step decomposition, such as least-to-most prompting
+(Zhou et al.,
+2022
+)
+, or use of external tools such as a scratchpad
+(Nye et al.,
+2021
+)
+or compiler
+(Gao et al.,
+2022
+)
+. Recently, CoT has been improved with search algorithms
+(Yao et al.,
+2023a
+; Hao et al.,
+2023
+; Besta et al.,
+2023
+)
+that can sample trajectories more effectively. Tree-of-thought (ToT) prompting
+(Yao et al.,
+2023a
+)
+uses DFS or BFS-based search guided by an LM-generated heuristic while Reasoning via Planning (RAP)
+(Hao et al.,
+2023
+)
+uses MCTS with rollouts simulated by the LM. However, they rely solely on LM internal knowledge and cannot adapt to useful external feedback.
+LLMs for acting.
+The strong reasoning and common-sense abilities of LLMs have also been adapted for decision-making or acting tasks as a policy model in interactive environments. In the realm of robotics LLMs have been employed as high-level controllers of control policies
+(Ahn et al.,
+2022
+; Huang et al.,
+2022
+; Driess et al.,
+2023
+)
+. Similar work
+(Baker et al.,
+2022
+; Wang et al.,
+2023
+; Zhu et al.,
+2023
+)
+has also adapted LLM agents to complex multimodal games such as Minecraft
+(Guss et al.,
+2019
+; Fan et al.,
+2022
+)
+. LLMs are particularly useful in text-based environments
+(Liu et al.,
+2018
+; Shridhar et al.,
+2020
+; Liu et al.,
+2023
+)
+, where acting-based prompting techniques such as ReAct
+(Yao et al.,
+2023b
+)
+have seen success. Similar to CoT, ReAct is limited by its simplicity and cannot effectively adapt to environment conditions. Many extensions have been proposed to address this, including Self-refine
+(Madaan et al.,
+2023
+)
+and Reflexion
+(Shinn et al.,
+2023
+; Yao et al.,
+2023c
+)
+, which uses self-reflection to enhance reasoning and decision-making, and AdaPlanner
+(Sun et al.,
+2023
+)
+, which incorporates both positive and negative environmental feedback. However these methods focus on refining an individual plan or trajectory and do not consider alternative choices at each step. In addition, recent work
+(Huang et al.,
+2023
+)
+has suggested LLMs cannot self-correct their internal reasoning, making it critical to use external feedback. Alternatively to pure decision-making environments, the reasoning and practical abilities of LLMs have been enhanced by access to external tools, such as APIs, search engines, calculators, or other models
+(Schick et al.,
+2023
+; Shen et al.,
+2023
+; Surís et al.,
+2023
+)
+. Contrary to reasoning-based approaches, these methods have not been improved with planning, limiting their effectiveness. We summarize them in Tab.
+1
+.
+Tree-based search.
+Tree-based search, where multiple branches of outcomes are explored during search, is widely used in many planning algorithms
+(Świechowski et al.,
+2023
+; LaValle et al.,
+2001
+)
+and Reinforcement Learning (RL)
+(Hafner et al.,
+2019
+; Du et al.,
+2023
+; Wu et al.,
+2023
+)
+algorithms for its good exploration-exploitation trade-off. Though tree-based search requires an environment model that can expand from arbitrary state
+(Vodopivec et al.,
+2017
+)
+, which often requires extra training in RL
+(Hafner et al.,
+2023
+)
+, such problem does not exist for LM tasks as we can conveniently backup to any state by setting the input to be the context and corresponding previous output by the LM. Thus, we work on the tree-based framework and use MCTS
+(Świechowski et al.,
+2023
+)
+to fully release the potential of LMs, while avoiding the cost of training a value function over language descriptions by leveraging the in-context learning
+(Brown et al.,
+2020
+)
+abilities of LLMs.
+3
+Preliminaries
+3.1
+Problem Setting and Prompting
+Before describing LATS, we first define our problem and outline a few established methods that leverage large language models for reasoning or decision-making. In LM reasoning or decision making, we are given an input
+x
+𝑥
+x
+in natural language and a pretrained language model
+p
+θ
+​
+(
+x
+)
+subscript
+𝑝
+𝜃
+𝑥
+p_{\theta}(x)
+parameterized by
+θ
+𝜃
+\theta
+; our goal is to generate a final output
+y
+∼
+p
+θ
+​
+(
+x
+)
+similar-to
+𝑦
+subscript
+𝑝
+𝜃
+𝑥
+y\sim p_{\theta}(x)
+corresponding to the answer (reasoning) or completes the task (decision-making). Both
+x
+𝑥
+x
+and
+y
+𝑦
+y
+are language
+sequences
+, which are comprised of a list of
+tokens
+(the basic elements of natural language, often words), denoted as
+x
+=
+(
+x
+​
+[
+1
+]
+,
+…
+,
+x
+​
+[
+n
+]
+)
+𝑥
+𝑥
+delimited-[]
+1
+…
+𝑥
+delimited-[]
+𝑛
+x=(x[1],\dots,x[n])
+and
+y
+=
+(
+y
+​
+[
+1
+]
+,
+…
+,
+y
+​
+[
+n
+]
+)
+𝑦
+𝑦
+delimited-[]
+1
+…
+𝑦
+delimited-[]
+𝑛
+y=(y[1],\dots,y[n])
+. The LM decodes text autoregressively, i.e., without other inputs, the probability for an LM to generate a sequence
+x
+𝑥
+x
+is given by
+p
+θ
+​
+(
+x
+)
+=
+∏
+i
+=
+1
+n
+p
+θ
+​
+(
+x
+​
+[
+i
+]
+|
+x
+​
+[
+1
+​
+…
+​
+i
+−
+1
+]
+)
+subscript
+𝑝
+𝜃
+𝑥
+superscript
+subscript
+product
+𝑖
+1
+𝑛
+subscript
+𝑝
+𝜃
+conditional
+𝑥
+delimited-[]
+𝑖
+𝑥
+delimited-[]
+1
+…
+𝑖
+1
+p_{\theta}(x)=\prod_{i=1}^{n}p_{\theta}(x[i]|x[1\dots i-1])
+. Usually, to improve the LM,
+prompts
+are provided along with the input
+x
+𝑥
+x
+, which are specific instructions or few-shot input-output examples. We denote the generic process where an input
+x
+𝑥
+x
+is transformed into an output
+y
+𝑦
+y
+by LM:
+y
+∼
+p
+θ
+​
+(
+y
+|
+prompt
+I
+​
+O
+​
+(
+x
+)
+)
+similar-to
+𝑦
+subscript
+𝑝
+𝜃
+conditional
+𝑦
+subscript
+prompt
+𝐼
+𝑂
+𝑥
+y\sim p_{\theta}(y|\texttt{prompt}_{IO}(x))
+, where
+prompt
+I
+​
+O
+​
+(
+x
+)
+subscript
+prompt
+𝐼
+𝑂
+𝑥
+\texttt{prompt}_{IO}(x)
+denotes the input
+x
+𝑥
+x
+.
+Chain-of-thought (CoT) Prompting
+(Wei et al.,
+2022
+)
+was introduced to cater to scenarios where direct mapping from
+x
+𝑥
+x
+to
+y
+𝑦
+y
+is intricate, such as when
+x
+𝑥
+x
+is from a mathematical query or challenging question. This method hinges on creating
+thoughts
+z
+1
+,
+…
+,
+z
+n
+subscript
+𝑧
+1
+…
+subscript
+𝑧
+𝑛
+z_{1},\dots,z_{n}
+that act as stepping stones between
+x
+𝑥
+x
+and
+y
+𝑦
+y
+; each thought
+z
+i
+subscript
+𝑧
+𝑖
+z_{i}
+is a language sequence. To employ CoT prompting, thoughts are extracted sequentially as
+z
+i
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+z
+i
+|
+x
+,
+z
+1
+​
+⋯
+​
+i
+−
+1
+)
+similar-to
+subscript
+𝑧
+𝑖
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+subscript
+𝑧
+𝑖
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑖
+1
+z_{i}\sim p_{\theta}^{CoT}(z_{i}|x,z_{1\cdots i-1})
+, with the final output being
+y
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+y
+|
+x
+,
+z
+1
+​
+⋯
+​
+n
+)
+similar-to
+𝑦
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+𝑦
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑛
+y\sim p_{\theta}^{CoT}(y|x,z_{1\cdots n})
+.
+Tree-of-thought (ToT) Prompting
+(Yao et al.,
+2023a
+)
+extends CoT prompting by exploring multiple reasoning paths over thoughts. It frames problems as a search over a tree where each node
+s
+=
+[
+x
+,
+z
+1
+⋅
+i
+]
+𝑠
+𝑥
+subscript
+𝑧
+⋅
+1
+𝑖
+s=[x,z_{1\cdot i}]
+represents a partial solution state comprising the original input
+x
+𝑥
+x
+and thought sequence
+z
+1
+​
+⋯
+​
+i
+subscript
+𝑧
+1
+⋯
+𝑖
+z_{1\cdots i}
+. Thoughts
+z
+i
+subscript
+𝑧
+𝑖
+z_{i}
+are generated by proposal or sampling with CoT
+z
+i
+∼
+p
+θ
+C
+​
+o
+​
+T
+​
+(
+z
+i
+|
+x
+,
+z
+1
+​
+⋯
+​
+i
+−
+1
+)
+similar-to
+subscript
+𝑧
+𝑖
+superscript
+subscript
+𝑝
+𝜃
+𝐶
+𝑜
+𝑇
+conditional
+subscript
+𝑧
+𝑖
+𝑥
+subscript
+𝑧
+1
+⋯
+𝑖
+1
+z_{i}\sim p_{\theta}^{CoT}(z_{i}|x,z_{1\cdots i-1})
+. Deliberate search algorithms like breadth-first or depth-first search are used to systematically explore the tree, guided by heuristics based on language model evaluations
+V
+​
+(
+s
+)
+𝑉
+𝑠
+V(s)
+of each state.
+Reasoning via Planning
+(RAP)
+(Hao et al.,
+2023
+)
+is similar to ToT, except that MCTS is used over DFS or BFS. Heuristics are designed from an LM, such as the likelihood or confidence of an action, and the LM is used as a world model to predict subsequent states during the simulation step.
+ReAct
+(Yao et al.,
+2023b
+)
+extends language models to tasks where the mapping from
+x
+𝑥
+x
+to
+y
+𝑦
+y
+is enhanced by or requires interactions with an external environment, such as a game or API. This technique constructs an action space
+A
+^
+=
+A
+∪
+Z
+^
+𝐴
+𝐴
+𝑍
+\hat{A}=A\cup Z
+that adds permissible actions
+a
+𝑎
+a
+to the reasoning traces
+z
+𝑧
+z
+from CoT. Observations
+o
+𝑜
+o
+from the environment are used to improve both reasoning and acting. To solve problems with ReAct, after each observation, actions are generated from
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+sequentially as
+a
+i
+∼
+p
+θ
+R
+​
+e
+​
+A
+​
+c
+​
+t
+​
+(
+a
+i
+|
+x
+,
+o
+1
+​
+⋯
+​
+i
+−
+1
+,
+a
+1
+​
+⋯
+​
+i
+−
+1
+)
+similar-to
+subscript
+𝑎
+𝑖
+superscript
+subscript
+𝑝
+𝜃
+𝑅
+𝑒
+𝐴
+𝑐
+𝑡
+conditional
+subscript
+𝑎
+𝑖
+𝑥
+subscript
+𝑜
+1
+⋯
+𝑖
+1
+subscript
+𝑎
+1
+⋯
+𝑖
+1
+a_{i}\sim p_{\theta}^{ReAct}(a_{i}|x,o_{1\cdots i-1},a_{1\cdots i-1})
+, with the final output being
+y
+∼
+p
+θ
+R
+​
+e
+​
+A
+​
+c
+​
+t
+​
+(
+y
+|
+x
+,
+o
+1
+​
+⋯
+​
+n
+,
+a
+1
+​
+⋯
+​
+n
+)
+similar-to
+𝑦
+superscript
+subscript
+𝑝
+𝜃
+𝑅
+𝑒
+𝐴
+𝑐
+𝑡
+conditional
+𝑦
+𝑥
+subscript
+𝑜
+1
+⋯
+𝑛
+subscript
+𝑎
+1
+⋯
+𝑛
+y\sim p_{\theta}^{ReAct}(y~{}|~{}x,o_{1\cdots n},a_{1\cdots n})
+.
+While the previously described prompting techniques improve LM performance on reasoning tasks, they falter on difficult tasks that involve multifaceted decision-making due to several shortcomings: 1)
+Flexibility
+: Base prompting methods (CoT or ReAct) autoregressively sample from the LM, neglecting potential alternative continuations from specific states. 2)
+Sensibility
+: Reasoning-based methods (CoT, RAP, or ToT) rely solely on the internal representations of the LM and cannot consider external observations. This dependency risks fact hallucination and error propagation while setting a performance ceiling. 3)
+Adaptability
+: Current planning frameworks (RAP or ToT) use simple search algorithms such as BFS or cannot leverage environmental feedback to improve planning. Additionally, the agent is static and cannot reuse previous experience or learn from trial and error. While RAP also adopts MCTS, it is constrained to tasks where the LM can become a world model and accurately predict states. These shortcomings limit the ability of LMs to be deployed as general problem-solving agents and form the motivation for LATS.
+3.2
+Monte-Carlo Tree Search (MCTS)
+Monte-Carlo Tree Search (MCTS) is a heuristic search algorithm that is proved successful on many decision-making environments such as Atari
+(Ye et al.,
+2021
+)
+and Go
+(Silver et al.,
+2016
+)
+. MCTS builds a decision tree where every node in the tree is a state and edge is an action. MCTS runs for
+k
+𝑘
+k
+episodes; for each episode, it starts from the root (i.e., initial state) and iteratively conducts two steps to expand the tree: 1)
+Expansion
+, where multiple children states
+s
+𝑠
+s
+are explored from the current parent state
+p
+𝑝
+p
+by sampling
+n
+𝑛
+n
+actions, and 2)
+Selection
+, where the children with the highest UCT
+(Upper Confidence bounds applied to Trees)
+(Kocsis & Szepesvári,
+2006
+)
+value is selected by the next iteration. The UCT of a child state
+s
+𝑠
+s
+is calculated as follows:
+U
+​
+C
+​
+T
+​
+(
+s
+)
+=
+V
+​
+(
+s
+)
++
+w
+​
+ln
+⁡
+N
+​
+(
+p
+)
+N
+​
+(
+s
+)
+,
+𝑈
+𝐶
+𝑇
+𝑠
+𝑉
+𝑠
+𝑤
+𝑁
+𝑝
+𝑁
+𝑠
+UCT(s)=V(s)+w\sqrt{\frac{\ln N(p)}{N(s)}},
+(1)
+where
+N
+​
+(
+s
+)
+𝑁
+𝑠
+N(s)
+is the number of visits to a node
+s
+𝑠
+s
+,
+V
+​
+(
+s
+)
+𝑉
+𝑠
+V(s)
+is the value function (expected return) from the subtree of
+s
+𝑠
+s
+,
+w
+𝑤
+w
+is the exploration weight, and
+p
+𝑝
+p
+is the parent node of
+s
+𝑠
+s
+. The child node with the highest UCT value is selected for expansion in the next iteration. When the end of an episode is reached, a
+backpropagation
+is carried out: the return
+r
+𝑟
+r
+is used for updating every
+V
+​
+(
+s
+)
+𝑉
+𝑠
+V(s)
+along the path
+with the formula
+V
+​
+(
+s
+)
+=
+V
+old
+​
+(
+s
+)
+​
+(
+N
+​
+(
+s
+)
+−
+1
+)
++
+r
+N
+​
+(
+s
+)
+𝑉
+𝑠
+subscript
+𝑉
+old
+𝑠
+𝑁
+𝑠
+1
+𝑟
+𝑁
+𝑠
+V(s)=\frac{V_{\text{old}}(s)(N(s)-1)+r}{N(s)}
+, where
+V
+old
+​
+(
+s
+)
+subscript
+𝑉
+old
+𝑠
+V_{\text{old}}(s)
+is the old value function. Normally, the major shortcoming of MCTS is that it requires an environment model to undo previous steps and form a searching tree, which is often a strong assumption. However, such a limitation does not exist for LMs, as we can conveniently reset to any step by simply copy-pasting historical text input. Such a special property is the key motivation of our work.
+4
+Unifying Planning, Reasoning, and Acting
+4.1
+LM Agent
+LATS supports sequential reasoning or decision-making tasks on the basis of ReAct. At time step
+t
+𝑡
+t
+, an agent receives an observation
+o
+t
+∈
+O
+subscript
+𝑜
+𝑡
+𝑂
+o_{t}\in O
+from the environment and takes an action
+a
+t
+∈
+A
+subscript
+𝑎
+𝑡
+𝐴
+a_{t}\in A
+following some policy
+π
+​
+(
+a
+t
+|
+x
+,
+o
+1
+​
+⋯
+​
+i
+−
+1
+,
+a
+1
+​
+⋯
+​
+i
+−
+1
+)
+𝜋
+conditional
+subscript
+𝑎
+𝑡
+𝑥
+subscript
+𝑜
+1
+⋯
+𝑖
+1
+subscript
+𝑎
+1
+⋯
+𝑖
+1
+\pi(a_{t}|x,o_{1\cdots i-1},a_{1\cdots i-1})
+, where
+x
+𝑥
+x
+consists of the task instruction and a number of few-shot examples. We initialize the agent with
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+to leverage the useful language representations of an LM as a base decision-maker. We follow the ReAct instantiation in which the action space
+A
+^
+=
+A
+∪
+Z
+^
+𝐴
+𝐴
+𝑍
+\hat{A}=A\cup Z
+consists of both the space of permissible actions
+A
+𝐴
+A
+and language space of reasoning traces
+Z
+𝑍
+Z
+. Actions directly affect the environment and result in observation, while thoughts are used to formalize decisions by organizing information, planning future actions, or injecting internal knowledge. The exact instantiation of the action space depends on the particular environment; for decision-making tasks actions might consist of commands on a website while for reasoning tasks the action space might be limited to a few external tools or APIs.
+Instead of greedily decoding one trajectory or solution, we sample
+n
+𝑛
+n
+actions from
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+using the current state. This is based on the intuition that for complex decision-making tasks, there is likely to be a range of potential trajectories or reasoning paths that are correct
+(Evans,
+2010
+)
+. Sampling a diverse set of candidates at each step mitigates the stochastic nature of LM text generation and enables greater exploration in both the decision-making and reasoning space. We wrap
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+within our proposed search algorithm to deliberately construct the best trajectory from sampled actions.
+4.2
+LATS
+Figure 3:
+An overview of the six operations of LATS. A node is
+selected
+,
+expanded
+,
+evaluated
+, then
+simulated
+until a terminal node is reached, then the resulting value is
+backpropagated
+. If the trajectory fails, a
+reflection
+is generated and used as additional context for future trials. These operations are performed in succession until the budget is reached or task is successful.
+The main component of LATS is a search algorithm that controls the overall problem-solving process with deliberate planning. To find the most promising trajectory and systemically balance exploration with exploitation, we adopt a variant of Monte Carlo Tree Search (MCTS) that frames decision-making as a tree search, in which each node
+s
+=
+[
+x
+,
+a
+1
+​
+⋯
+​
+i
+,
+o
+1
+​
+⋯
+​
+i
+]
+𝑠
+𝑥
+subscript
+𝑎
+1
+⋯
+𝑖
+subscript
+𝑜
+1
+⋯
+𝑖
+s=[x,a_{1\cdots i},o_{1\cdots i}]
+represents a state comprising the original input
+x
+𝑥
+x
+, action sequence
+a
+1
+⋅
+i
+subscript
+𝑎
+⋅
+1
+𝑖
+a_{1\cdot i}
+, and observation sequence
+o
+1
+⋅
+i
+subscript
+𝑜
+⋅
+1
+𝑖
+o_{1\cdot i}
+.
+To adapt MCTS for language agents, LATS repurposes
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+as an agent, state evaluator, and feedback generator, leveraging the useful language priors of modern LMs to facilitate planning. While standard MCTS and RAP
+Hao et al. (
+2023
+)
+rely on internal dynamics models to facilitate simulation, LATS is model-free and uses environment interaction. LATS consists of a series of operations,
+selection, expansion, evaluation, simulation, backpropagation, and reflection
+, performed in succession until the task is successfully completed or a computational limit is reached. The full psuedocode of LATS can be found in Sec.
+A
+in the Appendix.
+Selection.
+In the first operation, the algorithm identifies a segment of the current tree most suitable for subsequent expansion. Starting from the root node, denoted as the initial state
+s
+0
+subscript
+𝑠
+0
+s_{0}
+, a child node is selected at each tree level until a leaf node is reached. To balance exploration and exploitation, we use the UCT algorithm as shown in Eq.
+1
+.
+Expansion.
+After selecting a node, the second operation expands the tree by sampling
+n
+𝑛
+n
+actions from
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+, as described in the prior section. The environment receives each action and returns corresponding feedback as an observation. This results in
+n
+𝑛
+n
+new child nodes added to the tree. This tree is stored in an external long-term memory structure.
+Evaluation.
+The third operation assigns a scalar value to each new child node to be used for selection and backpropagation. This value effectively quantifies the agent’s progress in task completion, serving as a heuristic to steer the search algorithm towards the most promising regions of the tree. Following
+Yao et al. (
+2023a
+)
+we repurpose
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+into a value function by prompting it to reason about a given state. To obtain a scalar value, we instruct
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+to end its reasoning trace with a score indicating the correctness of the trajectory. This method offers enhanced flexibility over programmed heuristics
+(Campbell et al.,
+2002
+)
+and greater efficiency than learned heuristics
+(Silver et al.,
+2017
+)
+.
+Simulation.
+The fourth operation expands the currently selected node until a terminal state is reached. At each depth level we sample and evaluate nodes with the same operations, but prioritize nodes of highest value. Reaching a terminal state provides objective feedback on the correctness of a trajectory. If the task is completed successfully, then LATS terminates the search. If the solution is partially successful or unsuccessful, then we perform two additional operations as described below.
+Backpropagation.
+This operation updates the values of the tree based on the outcome of a trajectory. For each node
+s
+0
+,
+s
+1
+,
+…
+,
+s
+n
+subscript
+𝑠
+0
+subscript
+𝑠
+1
+…
+subscript
+𝑠
+𝑛
+s_{0},s_{1},\dots,s_{n}
+in the trajectory from root (initial state
+s
+0
+subscript
+𝑠
+0
+s_{0}
+) of the searching tree to leaf (terminal state
+s
+n
+subscript
+𝑠
+𝑛
+s_{n}
+), its value is updated to reflect the outcome of the simulation by
+N
+​
+(
+s
+i
+)
+=
+N
+old
+​
+(
+s
+i
+)
++
+1
+𝑁
+subscript
+𝑠
+𝑖
+subscript
+𝑁
+old
+subscript
+𝑠
+𝑖
+1
+N(s_{i})=N_{\text{old}}(s_{i})+1
+and
+V
+​
+(
+s
+i
+)
+=
+r
++
+N
+old
+​
+(
+s
+i
+)
+​
+V
+old
+​
+(
+s
+i
+)
+N
+​
+(
+s
+i
+)
+𝑉
+subscript
+𝑠
+𝑖
+𝑟
+subscript
+𝑁
+old
+subscript
+𝑠
+𝑖
+subscript
+𝑉
+old
+subscript
+𝑠
+𝑖
+𝑁
+subscript
+𝑠
+𝑖
+V(s_{i})=\frac{r+N_{\text{old}}(s_{i})V_{\text{old}}(s_{i})}{N(s_{i})}
+, where
+r
+𝑟
+r
+is the return and
+N
+old
+,
+V
+old
+subscript
+𝑁
+old
+subscript
+𝑉
+old
+N_{\text{old}},V_{\text{old}}
+are the old number of visits and value function. These updated values are used in the UCT formula (Eq.
+1
+) to guide the selection of the next node for exploration.
+Reflection.
+In addition to the environmental feedback, we also leverage
+self-reflection
+to further refine the decision-making process
+(Shinn et al.,
+2023
+; Madaan et al.,
+2023
+)
+. Upon encountering an unsuccessful terminal node,
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+is prompted with the trajectory and final reward to provide a verbal self-reflection that summarizes the errors in the reasoning or acting process and proposes superior alternatives. We store both failed trajectories and corresponding reflections in the memory. In subsequent iterations, these are integrated as additional context to the agent and value function, refining both through in-context learning. This imparts a semantic gradient signal more useful than a scalar value, enabling the agent to learn from trial and error without the cost of expensive optimization processes such as reinforcement learning.
+Conceptually, LATS has the following advantages as a general framework for reasoning and decision-making with LM agents.
+(1)
+Generality
+: LATS supports both reasoning and decision-making tasks by defining a shared space of thoughts and actions. (2)
+Deliberate
+: The use of MCTS and LM value function ensures a principled search that selects options with high value while exploring promising alternatives. (3)
+Adaptability
+: LATS is designed around the use of external feedback through observations and self-reflection, enabling greater adaptation during problem-solving. (4)
+Flexibility
+: LATS can accommodate different scenarios, environments, and resource stipulations by modifying state design and tree dimensions. (5)
+Modularity
+: The base LM agent, reflection generator, and value function can be independently altered and adapted to individual LM properties.
+5
+Experiments
+To demonstrate the general applicability of LATS, we evaluate our method on a variety of decision-making domains that requires both reasoning and acting ability: programming
+(Chen et al.,
+2021
+; Austin et al.,
+2021
+)
+, HotPotQA
+(Yang et al.,
+2018
+)
+, and WebShop
+(Yao et al.,
+2022
+)
+.
+5.1
+HotPotQA
+For a task that can be approached with both reasoning-based and acting-based strategies, we consider HotPotQA
+(Yang et al.,
+2018
+)
+, a multi-hop question-answering benchmark that requires retrieval over two or more Wikipedia passages. For the action space, in addition to LM thoughts we follow the setup from
+Yao et al. (
+2023b
+)
+, which provides the agent with API calls to search and lookup information. The output of these API calls and self-generated reflections form the observation space. We use a subset of 100 questions and three few-shot examples for each method. For ToT, we use DFS as the base search algorithm and scoring with the LM as the heuristic. For all methods that involve sampling, including LATS, we sample
+k
+=
+50
+𝑘
+50
+k=50
+trajectories. More details and prompts can be found in Sec.
+D
+and Sec.
+E
+in the Appendix.
+We evaluate internal reasoning strategies by removing actions and observations from the context, corresponding to CoT
+(Wei et al.,
+2022
+)
+and its variants, CoT-SC
+(Wang et al.,
+2022
+)
+, ToT
+(Yao et al.,
+2023a
+)
+, and RAP
+(Hao et al.,
+2023
+)
+. These methods rely solely on the agent’s existing knowledge to answer the question. We also consider acting-based methods ReAct, Reflexion, and LATS, which augment the agent with the interactive API environment and primarily evaluate its information retrieval abilities. While LATS is designed for scenarios where external feedback can enhance reasoning, we also implement a reasoning-only version with CoT as the base prompt. We also combine internal and external reasoning in LATS by first prompting with a CoT-based prompt, then switching to a ReAct-based prompt upon failure. This is closer to how humans might approach this task, by using tools to lookup additional information only when the answer is not already known.
+Prompt Method
+HotpotQA (EM)
+I/O
+0.32
+CoT
+(Wei et al.,
+2022
+)
+0.34
+CoT - SC
+(Wang et al.,
+2022
+)
+0.38
+ToT
+(Yao et al.,
+2023a
+)
+0.55
+RAP
+(Hao et al.,
+2023
+)
+0.60
+RAP (n = 10)
+0.60
+LATS (CoT)
+0.60
+Prompt Method
+HotpotQA (EM)
+ReAct
+(Yao et al.,
+2023b
+)
+0.32
+ReAct (best of k)
+0.38
+Reflexion
+(Shinn et al.,
+2023
+)
+0.51
+LATS
+0.61
+LATS (n = 3)
+0.56
+LATS (n = 10)
+0.64
+LATS (CoT + ReAct)
+0.71
+Table 2:
+GPT-3.5 reasoning-based prompting (left) and acting-based prompting (right) results on HotpotQA. LATS achieves the highest exact match (EM) for acting and is competitive on reasoning. Unless otherwise specified, we sample
+n
+=
+5
+𝑛
+5
+n=5
+nodes during expansion and
+k
+=
+50
+𝑘
+50
+k=50
+trajectories.
+Results.
+We observe in Tab.
+2
+that both internal reasoning and external retrieval strategies perform well on HotPotQA. Due to their large-scale training corpus, modern LLMs already encode factual knowledge and can often directly answer the question correctly. While CoT can slightly enhance performance on questions requiring reasoning, larger gains are observed with search methods ToT and RAP, which can sample and explore more outputs. We observe similar results for acting-based methods. LATS surpasses ReAct, even when sampling the same number of trajectories, by expanding more nodes with principled search (see Fig.
+5
+in Appendix
+D
+for a qualitative sample). This is demonstrated when modifying
+n
+𝑛
+n
+, the number of nodes expanded during each iteration. Increasing
+n
+𝑛
+n
+can consistently improve performance, although at greater computational and inference costs. LATS is also competitive to RAP on internal reasoning but performs worse than acting. Combining internal and external reasoning in LATS results in the highest performance, indicating the importance of external feedback in augmenting reasoning even in tasks the base LM can already perform.
+5.2
+Programming
+Prompt Method
+Model
+Pass@1
+CoT
+(Wei et al.,
+2022
+)
+GPT-3.5
+46.9
+ReAct
+(Yao et al.,
+2023b
+)
+GPT-3.5
+56.9
+Reflexion
+(Shinn et al.,
+2023
+)
+GPT-3.5
+68.1
+ToT
+(Yao et al.,
+2023a
+)
+GPT-3.5
+54.4
+RAP
+(Hao et al.,
+2023
+)
+GPT-3.5
+63.1
+LATS (Ours)
+GPT-3.5
+83.8
+I/O
+GPT-4
+80.1
+Reflexion
+GPT-4
+91.0
+LATS
+GPT-4
+94.4
+Prompt Method
+Pass@1
+CoT
+(Wei et al.,
+2022
+)
+54.9
+ReAct
+(Wei et al.,
+2022
+)
+67.0
+Reflexion
+(Shinn et al.,
+2023
+)
+70.0
+ToT
+(Yao et al.,
+2023a
+)
+65.8
+RAP
+(Hao et al.,
+2023
+)
+71.4
+LATS (Ours)
+81.1
+Table 3:
+GPT-3.5 and GPT-4 Pass@1 accuracy on HumanEval
+(Chen et al.,
+2021
+)
+and MBPP
+(Austin et al.,
+2021
+)
+. Prompting with LATS achieves the highest performance. We sample 5 solutions during expansion for
+8
+iterations.
+To demonstrate the importance of external observations for complex reasoning tasks, we evaluate the baselines and LATS on programming with Humaneval
+(Chen et al.,
+2021
+)
+and MBPP
+(Austin et al.,
+2021
+)
+. Both datasets measure the correctness of synthesized programs in Python from natural language docstrings. We use individual solutions as the action space and test suite and compiler feedback as the external observation. We follow
+Chen et al. (
+2022a
+)
+and use an LLM to generate a synthetic test suite of syntactically valid “assert” statements for each question. For each step, the solution is evaluated on this test suite, and the results including successful and failed tests and compiler output, are added to the context as an observation. We use the same test suite for Reflexion.
+For this task, the reasoning and acting baselines share an action space, but acting methods are able to incorporate observations as additional context. For LATS, since each action corresponds to a complete solution, we skip the simulation step of LATS and directly use the percentage of passed tests as the backpropagated reward. We use
+k
+=
+8
+𝑘
+8
+k=8
+iterations, set the number of generated tests at
+4
+4
+4
+, and sample
+n
+=
+5
+𝑛
+5
+n=5
+solutions during expansion. After the search is completed, we select the solution with the highest value and evaluate it on the real test suite for the pass@1 accuracy evaluation. More details and prompts can be found in Sec.
+D
+and Sec.
+F
+in the Appendix.
+Results.
+We find in Tab
+3
+that both search and semantic feedback are crucial for better performance. Despite not using observations, ToT and RAP are competitive with Reflexion. LATS has the highest performance on both datasets. Since RAP uses a similar search algorithm as LATS, this reveals the importance of external feedback for difficult reasoning tasks such as programming. With GPT-4, using LATS sets the state of the art for HumanEval, showing LATS can be used with more advanced LLMs for higher performance.
+5.3
+Webshop
+For a complex decision-making environment with practical applications, we consider WebShop
+(Yao et al.,
+2022
+)
+, an online shopping environment composed of a website with 1.18M real-world products and 12k human instructions. Agents must navigate a website through a variety of commands to purchase an item matching a user specification. We use the preconstructed action space of search and click commands and browser feedback and reflections for the observation. The performance is gauged using two metrics: an average score, reflecting the percentage of user-specified attributes met by the selected product, and a success rate, indicating the frequency with which the chosen product fulfills all given conditions. We compare against acting-based prompting methods and RL-based approaches. We evaluate on 50 instructions, expand
+n
+=
+5
+𝑛
+5
+n=5
+children for LATS, and set
+k
+=
+30
+𝑘
+30
+k=30
+for LATS, ReAct best of
+k
+𝑘
+k
+, and Reflexion. More details and prompts are in Appendix
+D
+and
+G
+.
+Results.
+We find in Tab.
+5
+that GPT-3.5 with ReAct is competitive to imitation learning, and can exceed reinforcement learning techniques with stronger prompting strategies. Sampling
+k
+=
+30
+𝑘
+30
+k=30
+trajectories with ReAct and Reflexion results in a similar performance, suggesting the semantic feedback is not as helpful in complex environments like WebShop. Indeed like in
+Shinn et al. (
+2023
+)
+, we find that generated reflections are often generic and do not provide useful feedback, resulting in a tendency for the agent to become stuck in local minima. However, using LATS indeed results in a noticeable improvement, indicating a more effective exploration for the same number of iterations.
+5.4
+Additional Observations
+Method
+Score
+SR
+ReAct
+(Yao et al.,
+2023b
+)
+53.8
+28.0
+ReAct (best of k)
+59.1
+32.0
+Reflexion
+(Shinn et al.,
+2023
+)
+64.2
+35.0
+LATS
+75.9
+38.0
+IL
+59.9
+29.1
+IL+RL
+62.4
+28.7
+Fine-tuning
+(Furuta et al.,
+2023
+)
+67.5
+45.0
+Expert
+82.1
+59.6
+Table 4:
+Score and success rate (SR) on Webshop. Table is separated into prompting, RL-based training, and human performance. For the same number of iterations, LATS improves both score and success rate, and surpasses RL-based training. IL/IL+RL taken from
+Yao et al. (
+2022
+)
+.
+Prompt Method
+HotPotQA (EM)
+ToT (ReAct)
+0.39
+RAP (ReAct)
+0.54
+LATS (No LM Heuristic)
+0.37
+LATS (DFS)
+0.42
+LATS (No Reflection)
+0.56
+LATS
+0.61
+Table 5:
+Ablation results on LATS and baseline variants in HotPotQA; we use ReAct as the base prompt and sample
+n
+=
+5
+𝑛
+5
+n=5
+children and
+k
+=
+50
+𝑘
+50
+k=50
+maximum trajectories. LATS requires every component and operation for optimal performance.
+We also conduct additional experiments on HotPotQA to demonstrate the effect of each component of LATS. We also design a version of ToT and RAP with ReAct prompt and can handle external observations. We use HotPotQA as our setup incorporates both reasoning (through thoughts) and acting (through API calls); the results are shown in Tab.
+5
+. More ablations for token consumption on HotPotQA are in Tab.
+7
+in Appendix
+C
+. Note that baselines generally perform worse than the reasoning-only setting of HotPotQA, which indicates that the acting-based setting is more challenging and adaption of search algorithms to decision-making scenarios is non-trivial.
+Self-reflection.
+We use self-reflection to provide additional semantic signals for the agent. We observe a
+0.05
+0.05
+0.05
+performance drop when removed from LATS, suggesting this is useful. This is a smaller gain Reflexion
+(Shinn et al.,
+2023
+)
+observes over ReAct
+(Yao et al.,
+2023b
+)
+as shown in Tab.
+2
+, suggesting overlap between the types of questions where there is an improvement with self-reflection and search. This variant outperforms RAP-ReAct, reflecting our improvements to MCTS.
+Search Algorithm.
+MCTS is a more principled search algorithm than variants like A* or DFS search and the basis for observed performance gains. We observe the effects of using DFS, and incorporate the LM-based heuristic used in ToT
+(Yao et al.,
+2023a
+)
+in which branches with low values are pruned. This removes the selection and backpropagation operations, and we observe a
+0.08
+0.08
+0.08
+drop in performance when sampling the same number of nodes, but outperforms ToT-ReAct.
+6
+Conclusion
+In this work, we introduce Language Agent Tree Search (LATS), the first framework to unify planning, acting, and reasoning for enhanced LLM problem solving. By deliberately constructing trajectories with search algorithms, incorporating external feedback, and enabling agents to learn from experience, LATS addresses key limitations of prior prompting techniques. Our evaluations demonstrate the ability of LATS to harness LLM capabilities for a variety of decision-making tasks while keeping its reasoning ability without additional training. The proposed synergies between search, interaction, and reflection offer a versatile approach to autonomous decision-making, highlighting the potential of LLMs as generalist agents. A full discussion of the limitations and broader impacts is in Appendix
+B
+.
+References
+Ahn et al. (2022)
+Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng.
+Do as i can, not as i say: Grounding language in robotic affordances.
+arXiv:2204.01691
+, 2022.
+Anthony et al. (2017)
+T. Anthony, Z. Tian, and D. Barber.
+Thinking fast and slow with deep learning and tree search.
+In
+NIPS
+, 2017.
+Austin et al. (2021)
+Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.
+Program synthesis with large language models.
+arXiv:2108.07732
+, 2021.
+Baker et al. (2022)
+Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune.
+Video pretraining (vpt): Learning to act by watching unlabeled online videos.
+arXiv:2206.11795
+, 2022.
+Besta et al. (2023)
+Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler.
+Graph of thoughts: Solving elaborate problems with large language models.
+arXiv:2308.09687
+, 2023.
+Bowman et al. (2015)
+Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning.
+A large annotated corpus for learning natural language inference.
+In
+EMNLP
+, 2015.
+Brown et al. (2020)
+Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
+Language models are few-shot learners.
+In
+NeurIPS
+, 2020.
+Campbell et al. (2002)
+Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu.
+Deep blue.
+Artificial intelligence
+, 2002.
+Chen et al. (2022a)
+Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen.
+Codet: Code generation with generated tests.
+arXiv:2207.10397
+, 2022a.
+Chen et al. (2021)
+Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.
+Evaluating large language models trained on code.
+arXiv:2107.03374
+, 2021.
+Chen et al. (2022b)
+Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen.
+Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.
+arXiv preprint arXiv:2211.12588
+, 2022b.
+Chowdhery et al. (2022)
+Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.
+Palm: Scaling language modeling with pathways.
+arXiv:2204.02311
+, 2022.
+Cobbe et al. (2021)
+Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
+Training verifiers to solve math word problems.
+arXiv:2110.14168
+, 2021.
+Deng et al. (2023)
+Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su.
+Mind2web: Towards a generalist agent for the web.
+arXiv:2306.06070
+, 2023.
+Driess et al. (2023)
+Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.
+Palm-e: An embodied multimodal language model.
+arXiv:2303.03378
+, 2023.
+Du et al. (2023)
+Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel.
+Learning universal policies via text-guided video generation.
+arXiv:2302.00111
+, 2023.
+Evans (2010)
+Jonathan St BT Evans.
+Intuition and reasoning: A dual-process perspective.
+Psychological Inquiry
+, 2010.
+Fan et al. (2022)
+Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar.
+Minedojo: Building open-ended embodied agents with internet-scale knowledge.
+In
+NeurIPS Datasets and Benchmarks Track
+, 2022.
+Furuta et al. (2023)
+Hiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur.
+Multimodal web navigation with instruction-finetuned foundation models.
+arXiv preprint arXiv:2305.11854
+, 2023.
+Gao et al. (2022)
+Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.
+Pal: Program-aided language models.
+arXiv preprint arXiv:2211.10435
+, 2022.
+Guo et al. (2018)
+Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang.
+Long text generation via adversarial training with leaked information.
+AAAI
+, 2018.
+Guss et al. (2019)
+William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov.
+Minerl: A large-scale dataset of minecraft demonstrations.
+In
+IJCAI
+, 2019.
+Hafner et al. (2019)
+Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.
+Learning latent dynamics for planning from pixels.
+In
+ICML
+, 2019.
+Hafner et al. (2023)
+Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.
+Mastering diverse domains through world models.
+arXiv:2301.04104
+, 2023.
+Hao et al. (2023)
+Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu.
+Reasoning with language model is planning with world model.
+arXiv:2305.14992
+, 2023.
+Huang et al. (2023)
+Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou.
+Large language models cannot self-correct reasoning yet.
+arXiv:2310.01798
+, 2023.
+Huang et al. (2022)
+Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al.
+Inner monologue: Embodied reasoning through planning with language models.
+arXiv:2207.05608
+, 2022.
+Jiang et al. (2018)
+D. Jiang, E. Ekwedike, and H. Liu.
+Feedback-based tree search for reinforcement learning.
+In
+ICML
+, 2018.
+Kocsis & Szepesvári (2006)
+Levente Kocsis and Csaba Szepesvári.
+Bandit based monte-carlo planning.
+In
+ECML
+, 2006.
+Kojima et al. (2022)
+Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
+Large language models are zero-shot reasoners.
+arXiv:2205.11916
+, 2022.
+LaValle et al. (2001)
+Steven M LaValle, James J Kuffner, BR Donald, et al.
+Rapidly-exploring random trees: Progress and prospects.
+Algorithmic and computational robotics: new directions
+, 2001.
+Liu et al. (2018)
+Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang.
+Reinforcement learning on web interfaces using workflow-guided exploration.
+In
+ICLR
+, 2018.
+Liu et al. (2023)
+Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang.
+Agentbench: Evaluating llms as agents.
+arXiv:2308.03688
+, 2023.
+Madaan et al. (2023)
+Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.
+Self-refine: Iterative refinement with self-feedback.
+arXiv:2303.17651
+, 2023.
+Nallapati et al. (2016)
+Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang.
+Abstractive text summarization using sequence-to-sequence rnns and beyond.
+In
+SIGNLL
+, 2016.
+Nye et al. (2021)
+Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al.
+Show your work: Scratchpads for intermediate computation with language models.
+arXiv:2112.00114
+, 2021.
+OpenAI (2023)
+OpenAI.
+Gpt-4 technical report.
+arXiv:2303.08774
+, 2023.
+Saparov & He (2022)
+Abulhair Saparov and He He.
+Language models are greedy reasoners: A systematic formal analysis of chain-of-thought.
+arXiv:2210.01240
+, 2022.
+Schick et al. (2023)
+Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
+Toolformer: Language models can teach themselves to use tools.
+arXiv:2302.04761
+, 2023.
+Shen et al. (2023)
+Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.
+Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.
+arXiv:2303.17580
+, 2023.
+Shinn et al. (2023)
+Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
+Reflexion: Language agents with verbal reinforcement learning.
+arXiv:2303.11366
+, 2023.
+Shridhar et al. (2020)
+Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht.
+Alfworld: Aligning text and embodied environments for interactive learning.
+arXiv:2010.03768
+, 2020.
+Silver et al. (2016)
+David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.
+Mastering the game of go with deep neural networks and tree search.
+nature
+, 2016.
+Silver et al. (2017)
+David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, L. Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis.
+Mastering the game of go without human knowledge.
+Nature
+, 2017.
+Sloman (1996)
+Steven A. Sloman.
+The empirical case for two systems of reasoning.
+Psychological Bulletin
+, 1996.
+Sun et al. (2023)
+Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang.
+Adaplanner: Adaptive planning from feedback with language models.
+arXiv:2305.16653
+, 2023.
+Surís et al. (2023)
+Dídac Surís, Sachit Menon, and Carl Vondrick.
+Vipergpt: Visual inference via python execution for reasoning.
+arXiv preprint arXiv:2303.08128
+, 2023.
+Świechowski et al. (2023)
+Maciej Świechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Mańdziuk.
+Monte carlo tree search: A review of recent modifications and applications.
+Artificial Intelligence Review
+, 2023.
+Touvron et al. (2023)
+Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and
+Thomas Scialom.
+Llama 2: Open foundation and fine-tuned chat models.
+arXiv:2307.09288
+, 2023.
+Vodopivec et al. (2017)
+Tom Vodopivec, Spyridon Samothrakis, and Branko Ster.
+On monte carlo tree search and reinforcement learning.
+Journal of Artificial Intelligence Research
+, 2017.
+Wang et al. (2023)
+Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
+Voyager: An open-ended embodied agent with large language models.
+arXiv:2305.16291
+, 2023.
+Wang et al. (2022)
+Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.
+Self-consistency improves chain of thought reasoning in language models.
+arXiv:2203.11171
+, 2022.
+Wei et al. (2022)
+Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.
+Chain of thought prompting elicits reasoning in large language models.
+arXiv:2201.11903
+, 2022.
+Wooldridge & Jennings (1995)
+Michael Wooldridge and Nicholas R Jennings.
+Intelligent agents: Theory and practice.
+The knowledge engineering review
+, 1995.
+Wu et al. (2023)
+Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg.
+Daydreamer: World models for physical robot learning.
+In
+CoRL
+. PMLR, 2023.
+Xie et al. (2023)
+Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie.
+Decomposition enhances reasoning via self-evaluation guided decoding.
+arXiv:2305.00633
+, 2023.
+Yang et al. (2018)
+Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
+Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
+arXiv:1809.09600
+, 2018.
+Yao et al. (2022)
+Shunyu Yao, Howard Chen, John Yang, and Karthik R Narasimhan.
+Webshop: Towards scalable real-world web interaction with grounded language agents.
+In
+NeurIPS
+, 2022.
+Yao et al. (2023a)
+Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.
+Tree of thoughts: Deliberate problem solving with large language models.
+arXiv:2305.10601
+, 2023a.
+Yao et al. (2023b)
+Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
+ReAct: Synergizing reasoning and acting in language models.
+In
+ICLR
+, 2023b.
+Yao et al. (2023c)
+Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese.
+Retroformer: Retrospective large language agents with policy gradient optimization.
+arXiv preprint arXiv:2308.02151
+, 2023c.
+Ye et al. (2021)
+Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao.
+Mastering atari games with limited data.
+In
+NeurIPS
+, 2021.
+Zhou et al. (2022)
+Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi.
+Least-to-most prompting enables complex reasoning in large language models.
+arXiv:2205.10625
+, 2022.
+Zhu et al. (2023)
+Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai.
+Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory.
+arXiv:2305.17144
+, 2023.
+7
+Appendix
+The appendix is organized as follows. First in Sec.
+A
+, we show the pseudocode of our proposed algorithm, LATS; then in Sec.
+B
+, we provide further discussion of our method and its limitations, future direction and broader impact; then in Sec.
+C
+we provide additional experimental results; then in Sec.
+D
+, we specify the environment details in our experiments; finally, we list our prompts used for the three environments in Sec.
+E
+(HotPotQA), Sec.
+F
+(Programming) and Sec.
+G
+(Webshop) respectively.
+Appendix A
+LATS Pseudocode
+Alg.
+1
+shows the pseudocode of our algorithm LATS. Nodes are stored explicitly in the memory. Unless otherwise specified, in all experiments we use
+n
+=
+5
+𝑛
+5
+n=5
+and
+w
+=
+1
+𝑤
+1
+w=1
+.
+Algorithm 1
+LATS
+⁡
+(
+S
+0
+,
+p
+θ
+,
+p
+V
+,
+p
+ref
+,
+d
+,
+k
+,
+n
+,
+w
+)
+LATS
+subscript
+𝑆
+0
+subscript
+𝑝
+𝜃
+subscript
+𝑝
+𝑉
+subscript
+𝑝
+ref
+𝑑
+𝑘
+𝑛
+𝑤
+\operatorname{LATS}(S_{0},p_{\theta},{p_{V}},p_{\text{ref}},d,k,n,w)
+Initial state
+s
+1
+subscript
+𝑠
+1
+s_{1}
+, action generator
+p
+θ
+subscript
+𝑝
+𝜃
+p_{\theta}
+, value function
+p
+V
+subscript
+𝑝
+𝑉
+p_{V}
+, reflection generator
+p
+ref
+subscript
+𝑝
+ref
+p_{\text{ref}}
+, number of generated actions
+n
+𝑛
+n
+, depth limit
+L
+𝐿
+L
+, number of roll-outs
+K
+𝐾
+K
+, context
+c
+𝑐
+c
+, and exploration weight
+w
+𝑤
+w
+Initialize action space
+A
+𝐴
+A
+, observation space
+O
+𝑂
+O
+Initialize the state-action value function
+p
+V
+:
+S
+×
+A
+↦
+ℝ
+:
+subscript
+𝑝
+𝑉
+maps-to
+𝑆
+𝐴
+ℝ
+{p_{V}}:S\times A\mapsto\mathbb{R}
+and visit counter
+N
+:
+S
+↦
+ℕ
+:
+𝑁
+maps-to
+𝑆
+ℕ
+{N}:S\mapsto\mathbb{N}
+to zero
+for
+k
+←
+0
+,
+…
+,
+K
+−
+1
+←
+𝑘
+0
+…
+𝐾
+1
+k\leftarrow 0,\dots,K-1
+do
+for
+t
+←
+0
+,
+…
+,
+L
+−
+1
+←
+𝑡
+0
+…
+𝐿
+1
+t\leftarrow 0,\dots,L-1
+do
+if
+s
+t
+subscript
+𝑠
+𝑡
+s_{t}
+not terminal
+then
+▷
+▷
+\triangleright
+Expansion & Simulation
+for
+i
+←
+1
+,
+…
+,
+n
+←
+𝑖
+1
+…
+𝑛
+i\leftarrow 1,\dots,n
+do
+Sample
+a
+t
+(
+i
+)
+∼
+p
+θ
+​
+(
+a
+∣
+s
+t
+)
+similar-to
+superscript
+subscript
+𝑎
+𝑡
+𝑖
+subscript
+𝑝
+𝜃
+conditional
+𝑎
+subscript
+𝑠
+𝑡
+a_{t}^{(i)}\sim p_{\theta}(a\mid s_{t})
+Get
+o
+t
+(
+i
+)
+superscript
+subscript
+𝑜
+𝑡
+𝑖
+o_{t}^{(i)}
+from environment,
+s
+t
++
+1
+(
+i
+)
+←
+(
+c
+t
+(
+i
+)
+,
+o
+t
+(
+i
+)
+,
+a
+t
+(
+i
+)
+)
+←
+superscript
+subscript
+𝑠
+𝑡
+1
+𝑖
+superscript
+subscript
+𝑐
+𝑡
+𝑖
+superscript
+subscript
+𝑜
+𝑡
+𝑖
+superscript
+subscript
+𝑎
+𝑡
+𝑖
+s_{t+1}^{(i)}\leftarrow(c_{t}^{(i)},o_{t}^{(i)},a_{t}^{(i)})
+,
+c
+t
++
+1
+(
+i
+)
+←
+(
+o
+t
+(
+i
+)
+,
+a
+t
+(
+i
+)
+)
+←
+superscript
+subscript
+𝑐
+𝑡
+1
+𝑖
+superscript
+subscript
+𝑜
+𝑡
+𝑖
+superscript
+subscript
+𝑎
+𝑡
+𝑖
+c_{t+1}^{(i)}\leftarrow(o_{t}^{(i)},a_{t}^{(i)})
+Evaluate
+V
+t
+(
+i
+)
+∼
+p
+V
+​
+(
+s
+t
+(
+i
+)
+)
+similar-to
+superscript
+subscript
+𝑉
+𝑡
+𝑖
+subscript
+𝑝
+𝑉
+superscript
+subscript
+𝑠
+𝑡
+𝑖
+{V}_{t}^{(i)}\sim{p_{V}}(s_{t}^{(i)})
+▷
+▷
+\triangleright
+Evaluation
+V
+​
+(
+s
+t
+)
+←
+V
+t
+(
+i
+)
+←
+𝑉
+subscript
+𝑠
+𝑡
+superscript
+subscript
+𝑉
+𝑡
+𝑖
+{V}(s_{t})\leftarrow{V}_{t}^{(i)}
+Add
+s
+t
+(
+i
+)
+superscript
+subscript
+𝑠
+𝑡
+𝑖
+s_{t}^{(i)}
+to children
+end
+for
+end
+if
+if
+s
+t
+subscript
+𝑠
+𝑡
+s_{t}
+is terminal
+then
+▷
+▷
+\triangleright
+Reflection
+Get
+r
+𝑟
+r
+from environment
+if
+r
+𝑟
+r
+not success
+then
+reflection
+←
+p
+ref
+​
+(
+c
+t
+)
+←
+reflection
+subscript
+𝑝
+ref
+subscript
+𝑐
+𝑡
+\text{reflection}\leftarrow p_{\text{ref}}(c_{t})
+c
+←
+reflection
+←
+𝑐
+reflection
+c\leftarrow\text{reflection}
+end
+if
+end
+if
+a
+t
+←
+arg
+⁡
+max
+a
+∈
+e
+​
+(
+s
+t
+)
+⁡
+[
+V
+​
+(
+s
+t
+)
++
+w
+​
+ln
+⁡
+N
+​
+(
+s
+t
+−
+1
+)
+N
+​
+(
+s
+t
+)
+]
+←
+subscript
+𝑎
+𝑡
+subscript
+𝑎
+𝑒
+subscript
+𝑠
+𝑡
+𝑉
+subscript
+𝑠
+𝑡
+𝑤
+𝑁
+subscript
+𝑠
+𝑡
+1
+𝑁
+subscript
+𝑠
+𝑡
+a_{t}\leftarrow\arg\max_{a\in e(s_{t})}\left[{V(s_{t})}+w\sqrt{\frac{\ln{N}(s_{t-1})}{{N}(s_{t})}}\right]
+▷
+▷
+\triangleright
+Selection
+N
+​
+(
+s
+t
++
+1
+)
+←
+N
+​
+(
+s
+t
++
+1
+)
++
+1
+←
+𝑁
+subscript
+𝑠
+𝑡
+1
+𝑁
+subscript
+𝑠
+𝑡
+1
+1
+{N}(s_{t+1})\leftarrow{N}(s_{t+1})+1
+if
+a
+t
+subscript
+𝑎
+𝑡
+a_{t}
+is an output action
+then
+break
+end
+for
+T
+←
+←
+𝑇
+absent
+T\leftarrow
+the actual number of steps
+for
+t
+←
+T
+−
+1
+,
+…
+,
+0
+←
+𝑡
+𝑇
+1
+…
+0
+t\leftarrow T-1,\dots,0
+do
+▷
+▷
+\triangleright
+Backpropagation
+V
+​
+(
+s
+t
+)
+←
+V
+​
+(
+s
+t
+)
+​
+(
+N
+​
+(
+s
+t
+)
+−
+1
+)
++
+r
+N
+​
+(
+s
+t
+)
+←
+𝑉
+subscript
+𝑠
+𝑡
+𝑉
+subscript
+𝑠
+𝑡
+𝑁
+subscript
+𝑠
+𝑡
+1
+𝑟
+𝑁
+subscript
+𝑠
+𝑡
+V(s_{t})\leftarrow\frac{V(s_{t})(N(s_{t})-1)+r}{N(s_{t})}
+end
+for
+end
+for
+Appendix B
+Discussion
+Limitations.
+Although LATS can improve reasoning and decision-making, this arrives at a higher computational cost relative to simpler prompting methods like ReAct or Reflexion. The search process takes more time than standard prompting or simpler techniques, and requires greater inference costs. While such an issue is mitigated by the fact that the number of nodes
+n
+𝑛
+n
+expanded at every step provides a natural trade-off between performance and efficiency (setting
+n
+=
+1
+𝑛
+1
+n=1
+makes the method as effecient as ReAct with multiple trials or CoT-SC), in practice we recommend using LATS for difficult tasks like programming or for situations where performance is prioritized over efficiency. We hope that continued advancements in LLMs will reduce costs and increase the practicality of LATS.
+Additionally, the benchmarks we use in this paper are relatively simple and focused on decision-making, compared to the complexity of real-world interactive environments. In addition, some environments might not easily support rollbacks to previous states. However, the design of LATS is flexible and can be adjusted to various resource constraints. Using planning-based prompting methods like LATS in environments like Minecraft
+(Fan et al.,
+2022
+)
+and more reasoning benchmarks would be interesting avenues for future work.
+Broader impact.
+LATS is a framework that enhances LLM performance through interactions with an environment. This improvement in autonomous decision-making may facilitate harmful uses of LLMs. Alternatively, LATS enhances interpretability and the potential for greater alignment, as it generates understandable, high-level linguistic reasoning and actions through several rounds of decision-making and reflection, rather than relying on implicit, low-level token values.
+Appendix C
+Ablations
+Prompt Method
+HotpotQA (EM)
+LATS (w=0.5)
+0.55
+LATS (w=2.0)
+0.61
+LATS (d=4)
+0.58
+LATS (CoT)
+0.60
+LATS (No LM Heuristic)
+0.37
+LATS
+0.61
+Table 6:
+Ablation results on LATS and baseline variants in HotPotQA measured by Exact Match (EM). We test different depth
+d
+𝑑
+d
+, exploration factor
+w
+𝑤
+w
+, and versions of LATS using CoT and without the LM value function. We sample
+n
+=
+5
+𝑛
+5
+n=5
+and
+k
+=
+50
+𝑘
+50
+k=50
+trajectories.
+Figure 4:
+Performance over successive iterations on HumanEval with GPT-3.5.
+In this section, we ablate various designs of LATS. Experiments are conducted on HotPotQA with a maximum of
+k
+=
+50
+𝑘
+50
+k=50
+trajectories and sampling size of
+n
+=
+5
+𝑛
+5
+n=5
+and HumanEval with a maximum of
+k
+=
+8
+𝑘
+8
+k=8
+trajectories and sampling size of
+n
+=
+5
+𝑛
+5
+n=5
+. The result for HotPotQA is shown in Tab.
+5
+and HumanEval in Fig.
+4
+.
+Exploration weight.
+We find that there is lower performance on HotPotQA when the exploration weight
+w
+𝑤
+w
+in the selection formula is decreased to
+0.5
+0.5
+0.5
+, suggesting that this reduces the effectiveness of the search. Increasing
+w
+𝑤
+w
+to
+2.0
+2.0
+2.0
+does not lead to a performance improvement, but we tend to observe faster convergence. The optimal setting depends on the particular environment and complexity of the state space.
+Depth.
+In our main experiments we use a maximum depth of
+d
+=
+7
+𝑑
+7
+d=7
+on HotPotQA for all methods, following previous work
+(Yao et al.,
+2023b
+)
+. We ablate the effect on LATS after reducing it to
+d
+=
+4
+𝑑
+4
+d=4
+. This results in only a slight drop in performance. We find that most questions can be answered within four steps, and using a greater number of steps tends to force the agent into local minima and rarely improves success.
+LM value function.
+The LM value function scores states based on expected future reward. Without this heuristic, the only signal to guide search would be from environment rewards for completed trajectories, which are scarce and often binary. When we remove the evaluation operation, we observe a dramatic
+0.24
+0.24
+0.24
+drop in performance.
+Performance over time.
+To see the effects of increasing the number of trajectories sampled, we change
+k
+𝑘
+k
+to different values. We conduct this experiment on HumanEval, which has a more noticeable difference due to sampling less trajectories. The results are shown in Fig.
+4
+, in which LATS scales better with more iterations than Reflexion.
+Sample complexity and Token cost.
+One possible concern of LATS is that the tree-structured search might consume much more tokens than existing methods. To further study the computational cost of LATS compared to prior methods, we examine the sample complexity (i.e. asymptotic token cost) of all methods considered in this paper, and count the average number of nodes expanded by our method and other tree-structured methods (ToT and RAP) upon successful search on HotPotQA. We present the results in Tab.
+7
+; the result shows that our method has the same sample complexity as other tree-based search methods, and has less average number of nodes expanded upon success, which indicates less token cost. The token cost gap will be even larger when taking failed trajectories into account, since our method has higher success rate and reaches computational budget limit less often.
+Method
+Performance (
+↑
+↑
+\uparrow
+)
+Sample complexity (
+↓
+↓
+\downarrow
+)
+Avg. #nodes upon success (
+↓
+↓
+\downarrow
+)
+ReAct (Best
+k
+=
+250
+𝑘
+250
+k=250
+)
+0.42
+0.42
+0.42
+O
+​
+(
+k
+)
+𝑂
+𝑘
+O(k)
+N/A
+CoT-SC (
+n
+=
+1
+,
+k
+=
+250
+formulae-sequence
+𝑛
+1
+𝑘
+250
+n=1,k=250
+)
+0.40
+0.40
+0.40
+O
+​
+(
+k
+)
+𝑂
+𝑘
+O(k)
+N/A
+LATS (
+n
+=
+1
+,
+k
+=
+50
+formulae-sequence
+𝑛
+1
+𝑘
+50
+n=1,k=50
+)
+0.48
+0.48
+0.48
+O
+​
+(
+k
+)
+𝑂
+𝑘
+O(k)
+N/A
+ToT (ReAct)
+0.49
+0.49
+0.49
+O
+​
+(
+k
+​
+n
+)
+𝑂
+𝑘
+𝑛
+O(kn)
+84.05
+84.05
+84.05
+RAP (ReAct)
+0.54
+0.54
+0.54
+O
+​
+(
+k
+​
+n
+)
+𝑂
+𝑘
+𝑛
+O(kn)
+70.60
+70.60
+70.60
+LATS (
+n
+=
+5
+,
+k
+=
+50
+formulae-sequence
+𝑛
+5
+𝑘
+50
+n=5,k=50
+)
+0.61
+0.61
+0.61
+O
+​
+(
+k
+​
+n
+)
+𝑂
+𝑘
+𝑛
+O(kn)
+66.65
+66.65
+66.65
+Table 7:
+The performance, sample complexity of different methods and average number of nodes expanded upon success by methods with tree-based search.
+n
+𝑛
+n
+is the number of children nodes expanded at every step and
+k
+𝑘
+k
+is the number of trajectories. Our method has the same sample complexity as other methods with tree-based search and expands less nodes upon success, which indicates lower token cost.
+Appendix D
+Environment Details
+D.1
+HotPotQA
+Figure 5:
+Example trajectories on HotPotQA for ReAct (left) and LATS (right). LATS can sample more actions and avoid failure from previous mistakes by evaluating states with an LM to guide the search toward promising areas of the tree.
+HotPotQA
+(Yang et al.,
+2018
+)
+is a question-answering dataset that requires reasoning over multiple supporting documents to answer questions. It contains 113k Wikipedia-based question-answer pairs crafted by crowdworkers to be diverse, multi-hop, and explainable. Questions cover a range of types like entities, locations, dates, and comparison of shared properties between two entities. Crowdworkers also provide supporting facts from the documents that justify the answer. We use the HotPotQA benchmark setting with all the Wikipedia paragraphs to test retrieval. We use a randomly selected subset of 100 questions for our experiments and a maximum depth limit of 6. Fig.
+5
+illustrates how ReAct and LATS work on an example task of HotPotQA, and gives a qualitative example on how LATS outperforms ReAct on the task.
+Action Space.
+We adopt the Wikipedia web API proposed in
+Yao et al. (
+2023b
+)
+, with three types of actions to support interactive information retrieval:
+(1)
+search
+[
+entity
+], which returns the first 5 sentences from the corresponding
+entity
+wiki page if it exists, or else suggests top-5 similar entities from the Wikipedia search engine,
+(2)
+lookup
+[
+string
+], which returns the next sentence in the page containing
+string
+,
+(3)
+finish
+[
+answer
+], which finishes the current task with
+answer
+.
+These API calls and free-form thoughts form the action space for this environment.
+D.2
+Programming
+The HumanEval dataset
+(Chen et al.,
+2021
+)
+is a collection of 164 handwritten programming problems introduced to evaluate the functional correctness of models for synthesizing programs from natural language descriptions. Each problem includes a function signature, docstring description, reference implementation, and multiple unit tests, with an average of 7.7 tests per problem. The programming tasks assess comprehension of natural language, reasoning, algorithms, and basic mathematics, at a difficulty level comparable to simple software interview questions. Pass rates are evaluated with the pass@k metric, where k samples are generated per problem and a problem is considered solved if any sample passes all tests. We use all 164 problems for our experiments and a maximum depth limit of 8.
+The Mostly Basic Programming Problems (MBPP)
+Austin et al. (
+2021
+)
+benchmark contains 974 short Python functions designed to evaluate program synthesis techniques. The dataset was constructed by crowdsourcing from workers with basic Python knowledge. Each data point consists of a natural language description of a programming task, a reference solution implementation, and three test cases for functional correctness. The natural language prompts are typically short, one-sentence descriptions. Solutions cover common programming constructs including mathematical operations, list processing, string manipulation, and usage of the Python standard library. On average, solutions are 6.8 lines of code. The dataset is also supplemented with an additional set of 426 problems that were manually verified for unambiguous specifications, standard function signatures, and accurate test cases. We use a randomly selected subset of 397 problems for our experiments.
+D.3
+WebShop
+WebShop
+(Yao et al.,
+2022
+)
+is an interactive web-based environment designed to evaluate agents on grounded language understanding and decision-making. It simulates an e-commerce shopping task by providing agents with over 1 million real-world products scraped from Amazon, spanning 5 categories and 113 subcategories. These products contain rich linguistic information, with an average text length of 262 words and a vocabulary size of 224k. In addition, there are over 800k unique product options available for customization. The environment renders webpages in two modes: HTML mode provides pixel-level observations with interactive elements, while simple mode converts the raw HTML into a structured text observation more amenable for training agents. The action space consists of query searches and button clicks, which transition between 4 page types: search, results, item and item-detail. Instructions are crowdsourced natural language specifying product attributes and options, with a total of 12k collected. Automatic rewards are computed by comparing the product purchased by the agent against the attributes and options specified in the instruction, using both lexical matching and semantic similarity metrics.
+Type
+Argument
+State
+→
+→
+\rightarrow
+Next State
+search
+[
+Query
+]
+Search
+→
+→
+\rightarrow
+Results
+choose
+Back to search
+∗
+*
+→
+→
+\rightarrow
+Search
+choose
+Prev/Next page
+Results
+→
+→
+\rightarrow
+Results
+choose
+[
+Product title
+]
+Results
+→
+→
+\rightarrow
+Item
+choose
+[
+Option
+]
+Item
+→
+→
+\rightarrow
+Item
+choose
+Desc/Overview
+Item
+→
+→
+\rightarrow
+Item-Detail
+choose
+Previous
+Item-Detail
+→
+→
+\rightarrow
+Item
+choose
+Buy
+Item
+→
+→
+\rightarrow
+Episode End
+Table 8:
+Action space of webshop.
+There are two evaluation metrics used in WebShop: (1)
+Task Score
+: defined as
+(
+100
+×
+avg. reward
+)
+100
+avg. reward
+(100\times\text{avg. reward})
+, which captures the average reward obtained across episodes; and (2)
+Success Rate (SR)
+defined as the portion of instructions where
+r
+=
+1
+𝑟
+1
+r=1
+. The reward is calculated based on the number of attributes satisfied by the selected item. We use 50 environments for our experiments and a maximum depth limit of 15.
+Appendix E
+HotPotQA Prompts
+E.1
+Base Acting Prompt
+Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types:
+(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
+(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
+(3) Finish[answer], which returns the answer and finishes the task.
+After each observation, provide the next Thought and next Action.
+Here are some examples:
+Question: Which magazine was started first Arthur’s Magazine or First for Women?
+Thought 1: I need to search Arthur’s Magazine and First for Women, and find which was started first.
+Action 1: Search[Arthur’s Magazine]
+Observation 1: Arthur’s Magazine (18440̆0e20̆0800̆0931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey’s Lady’s Book.[3]
+Thought 2: Arthur’s Magazine was started in 1844. I need to search First for Women next.
+Action 2: Search[First for Women]
+Observation 2: First for Women is a woman’s magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey.[2] In 2011 the circulation of the magazine was 1,310,696 copies.[3]
+Thought 3: First for Women was started in 1989. 1844 (Arthur’s Magazine) ¡ 1989 (First for Women), so Arthur’s Magazine was started first.
+Action 3: Finish[Arthur’s Magazine]
+(examples)
+You have attempted to answer the following question before and failed. The following reflection(s) give a plan to avoid failing to answer the question in the same way you did previously. Use them to improve your strategy of correctly answering the given question.
+(trajectories)
+(input)
+E.2
+Base Reasoning Prompt
+Solve a question answering task by having a Thought, then Finish with your answer. Thought can reason about the current situation. Finish[answer] returns the answer and finishes the task. You will be given context that you should use to help you answer the question. Start your response with either Action or an indexed Thought
+Here are some examples:
+Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?
+Let’s think step by step.
+Thought 1: The eastern sector of Colorado orogeny extends into the High Plains.
+Thought 2: High Plains rise in elevation from around 1,800 to 7,000 ft
+Thought 3: The answer is 1,800 to 7,000 ft.
+Action: Finish[1,800 to 7,000 ft]
+(examples)
+Previous trial:
+(trajectories)
+(input)
+E.3
+Value Function Prompt
+Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by environmental observations about the situation, thoughts that can reason about the current situation and actions that can be three types:
+(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
+(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
+(3) Finish[answer], which returns the answer and finishes the task.
+Given a question and a trajectory, evaluate its correctness and provide your reasoning and analysis in detail. Focus on the latest thought, action, and observation. Incomplete trajectories can be correct if the thoughts and actions so far are correct, even if the answer is not found yet. Do not generate additional thoughts or actions. Then at the last line conclude ”Thus the correctness score is s”, where s is an integer from 1 to 10.
+Question: Which magazine was started first Arthur’s Magazine or First for Women?
+Thought 1: I need to search Arthur’s Magazine and First for Women, and find which was started first.
+Action 1: Search[Arthur’s Magazine]
+Observation 1: Arthur’s Magazine (18440̆0e20̆0800̆0931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey’s Lady’s Book.[3]
+This trajectory is correct as it is reasonable to search for the first magazine provided in the question. It is also better to have simple searches corresponding to a single entity, making this the best action.
+Thus the correctness score is 10
+(other examples)
+(failed trajectories)
+(context)
+E.4
+Reflection Prompt
+Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by environmental observations about the situation, thoughts that can reason about the current situation and actions that can be three types:
+(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
+(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
+(3) Finish[answer], which returns the answer and finishes the task.
+Given a question and a trajectory, evaluate its correctness and provide your reasoning and analysis in detail. Focus on the latest thought, action, and observation. Incomplete trajectories can be correct if the thoughts and actions so far are correct, even if the answer is not found yet. Do not generate additional thoughts or actions. Then at the last line conclude ”Thus the correctness score is s”, where s is an integer from 1 to 10.
+Question: Which magazine was started first Arthur’s Magazine or First for Women?
+Thought 1: I need to search Arthur’s Magazine and First for Women, and find which was started first.
+Action 1: Search[Arthur’s Magazine]
+Observation 1: Arthur’s Magazine (18440̆0e20̆0800̆0931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey’s Lady’s Book.[3]
+This trajectory is correct as it is reasonable to search for the first magazine provided in the question. It is also better to have simple searches corresponding to a single entity, making this the best action.
+Thus the correctness score is 10
+(other examples)
+(failed trajectories)
+(context)
+Appendix F
+Programming Prompts
+F.1
+HumanEval function implementation example
+Sample function signature:
+⬇
+def
+minSubArraySum
+(
+nums
+):
+Given
+an
+array
+of
+integers
+nums
+,
+find
+the
+minimum
+sum
+of
+any
+non
+-
+empty
+sub
+-
+array
+of
+nums
+.
+Example
+minSubArraySum
+([2,
+3,
+4,
+1,
+2,
+4])
+==
+1
+minSubArraySum
+([-1,
+-2,
+-3])
+==
+-6
+Sample function body implementation:
+⬇
+min_sum
+=
+float
+(’
+inf
+’)
+for
+i
+in
+range
+(
+len
+(
+nums
+)):
+current_sum
+=
+0
+for
+j
+in
+range
+(
+i
+,
+len
+(
+nums
+)):
+current_sum
++=
+nums
+[
+j
+]
+if
+current_sum
+<
+min_sum
+:
+min_sum
+=
+current_sum
+return
+min_sum
+F.2
+Base Acting/Reasoning Prompt
+You are an AI Python assistant. You will be given your previous implementation of a function, a series of unit tests results, and your self-reflection on your previous implementation. Write your full implementation (restate the function signature).
+Example 1:
+[previous impl]:
+⬇
+def
+add
+(
+a
+:
+int
+,
+b
+:
+int
+)
+->
+int
+:
+”””
+Given
+integers
+a
+and
+b
+,
+return
+the
+total
+value
+of
+a
+and
+b
+.
+”””
+return
+a
+-
+b
+[unit test results from previous impl]:
+Tested passed:
+Tests failed:
+assert add(1, 2) == 3 # output: -1
+assert add(1, 2) == 4 # output: -1
+[reflection on previous impl]:
+The implementation failed the test cases where the input integers are 1 and 2. The issue arises because the code does not add the two integers together, but instead subtracts the second integer from the first. To fix this issue, we should change the operator from ‘-‘ to ‘+‘ in the return statement. This will ensure that the function returns the correct output for the given input.
+[improved impl]:
+⬇
+def
+add
+(
+a
+:
+int
+,
+b
+:
+int
+)
+->
+int
+:
+”””
+Given
+integers
+a
+and
+b
+,
+return
+the
+total
+value
+of
+a
+and
+b
+.
+”””
+return
+a
++
+b
+F.3
+Reflection Prompt
+You are a Python programming assistant. You will be given a function implementation and a series of unit test results. Your goal is to write a few sentences to explain why your implementation is wrong as indicated by the tests. You will need this as guidance when you try again later. Only provide the few sentence description in your answer, not the implementation. You will be given a few examples by the user.
+Example 1:
+[previous impl]:
+⬇
+def
+add
+(
+a
+:
+int
+,
+b
+:
+int
+)
+->
+int
+:
+”””
+Given
+integers
+a
+and
+b
+,
+return
+the
+total
+value
+of
+a
+and
+b
+.
+”””
+return
+a
+-
+b
+[unit test results from previous impl]:
+Tested passed:
+Tests failed:
+assert add(1, 2) == 3 # output: -1
+assert add(1, 2) == 4 # output: -1
+[reflection on previous impl]:
+The implementation failed the test cases where the input integers are 1 and 2. The issue arises because the code does not add the two integers together, but instead subtracts the second integer from the first. To fix this issue, we should change the operator from ‘-‘ to ‘+‘ in the return statement. This will ensure that the function returns the correct output for the given input.
+F.4
+Test Case Generation Prompt
+You are an AI coding assistant that can write unique, diverse, and intuitive unit tests for functions given the signature and docstring.
+(input)
+Appendix G
+WebShop Prompts
+G.1
+Acting Prompt
+Webshop
+Instruction:
+i would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars
+[Search]
+Action: search[3 ounce bright citrus deodorant sensitive skin]
+Observation:
+[Back to Search]
+Page 1 (Total results: 50)
+[Next
+]
+[B078GWRC1J]
+Bright Citrus Deodorant by Earth Mama — Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
+$10.99
+[B078GTKVXY]
+Ginger Fresh Deodorant by Earth Mama — Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
+$10.99
+[B08KBVJ4XN]
+Barrel and Oak - Aluminum-Free Deodorant, Deodorant for Men, Essential Oil-Based Scent, 24-Hour Odor Protection, Cedar & Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
+$15.95
+Action: think[B078GWRC1J and B078GTKVXY are bright citrus deodorant less then 50 dollars. I can check B078GWRC1J first.]
+Observation: OK.
+Action: click[B078GWRC1J]
+Observation:
+[Back to Search]
+[Prev]
+scent [assorted scents][bright citrus][calming lavender][ginger fresh][simply non-scents]
+size [travel set (4-pack)][3 ounce (pack of 1)][3-ounce (2-pack)]
+Bright Citrus Deodorant by Earth Mama — Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
+Price: $10.99
+Rating: N.A.
+[Description]
+[Features]
+[Reviews]
+[Buy Now]
+Action: think[For 3 ounce bottle of bright citrus deodorant for sensitive skin, the item has options ’bright citrus’ and ’3 ounce (pack of 1)’ and seems good to buy.]
+Observation: OK.
+Action: click[bright citrus]
+Observation: You have clicked bright citrus.
+Action: click[3 ounce (pack of 1)]
+Observation: You have clicked 3 ounce (pack of 1).
+Action: click[Buy Now]
+G.2
+Value Function Prompt
+Given an item to purchase and a trajectory that aims to buy an item that exactly matches the specification, which corresponds to the ideal score of 1.0, analyze the following trajectory, then at the last line conclude ”Thus the correctness score is s”, where s is an integer from 1 to 10.
+Here are some examples
+Webshop
+Instruction:
+i am looking for dairy free and apple variety pack of chips, and price lower than 30.00 dollars
+[Search]
+Action: search[dairy free and apple variety pack of chips]
+Observation:
+[Back to Search]
+Page 1 (Total results: 50)
+[Next]
+[B07HRFJWP8]
+Enjoy Life Foods Soft Baked Ovals, Breakfast Bars, Nut Free Bars, Soy Free, Dairy Free, Non GMO, Gluten Free, Vegan, Variety Pack, 4 Boxes (20 Bars Total)
+$100.0
+[B01KMHY5PG]
+Enjoy Life Soft Baked Chewy Bars, Variety Pack, Nut Free Bars, Soy Free, Dairy Free, Gluten Free, 6 Boxes (30 Total Bars)
+$21.49
+[B008D2X8C4]
+Enjoy Life Lentil Chips Variety Pack, Dairy Free Chips, Soy Free, Nut Free, Non GMO, Vegan, Gluten Free, 24 Bags (0.8 oz)
+$100.0
+Action: think[The first two items are not what I want. The third item is the dairy free and apple variety pack of chips I want.]
+Observation: OK.
+Action: click[B008D2X8C4]
+Observation:
+[Back to Search]
+[Prev]
+flavor name [dill and sour cream][garlic & parmesan][light sea salt][margherita pizza][thai chili lime][variety pack]
+size [0.8 ounce (pack of 24)][4 ounce (pack of 12)]
+Price: $100.0
+Rating: N.A.
+[Description]
+[Features]
+[Reviews]
+[Buy Now]
+Action: think[The item has options ’variety pack’ and ’0.8 ounce (pack of 24)’ and seems good to buy.]
+Observation: OK.
+Action: click[variety pack]
+Observation: You have clicked variety pack.
+Action: click[0.8 ounce (pack of 24)]
+Observation: You have clicked 0.8 ounce (pack of 24).
+Action: click[Buy Now]
+Reflection: In this attempt, I was unsuccessful. I accidentally bought a product that was $100, which is more than my budget of $30. Either way, the initial search results were not good. Next time, I will do search[”variety pack of chips”] and then check if the results meet the dairy free and the $30 budget constraints. I will continue to refine my searches so that I can find more products.
+Thus the correctness score is 5
+(input)
+G.3
+Reflection Prompt
+Given an item to purchase and a trajectory that aims to buy an item that exactly matches the specification, which corresponds to the ideal score of 1.0, analyze the following trajectory, then at the last line conclude ”Thus the correctness score is s”, where s is an integer from 1 to 10.
+Here are some examples
+Webshop
+Instruction:
+i am looking for dairy free and apple variety pack of chips, and price lower than 30.00 dollars
+[Search]
+Action: search[dairy free and apple variety pack of chips]
+Observation:
+[Back to Search]
+Page 1 (Total results: 50)
+[Next]
+[B07HRFJWP8]
+Enjoy Life Foods Soft Baked Ovals, Breakfast Bars, Nut Free Bars, Soy Free, Dairy Free, Non GMO, Gluten Free, Vegan, Variety Pack, 4 Boxes (20 Bars Total)
+$100.0
+[B01KMHY5PG]
+Enjoy Life Soft Baked Chewy Bars, Variety Pack, Nut Free Bars, Soy Free, Dairy Free, Gluten Free, 6 Boxes (30 Total Bars)
+$21.49
+[B008D2X8C4]
+Enjoy Life Lentil Chips Variety Pack, Dairy Free Chips, Soy Free, Nut Free, Non GMO, Vegan, Gluten Free, 24 Bags (0.8 oz)
+$100.0
+Action: think[The first two items are not what I want. The third item is the dairy free and apple variety pack of chips I want.]
+Observation: OK.
+Action: click[B008D2X8C4]
+Observation:
+[Back to Search]
+[Prev]
+flavor name [dill and sour cream][garlic & parmesan][light sea salt][margherita pizza][thai chili lime][variety pack]
+size [0.8 ounce (pack of 24)][4 ounce (pack of 12)]
+Price: $100.0
+Rating: N.A.
+[Description]
+[Features]
+[Reviews]
+[Buy Now]
+Action: think[The item has options ’variety pack’ and ’0.8 ounce (pack of 24)’ and seems good to buy.]
+Observation: OK.
+Action: click[variety pack]
+Observation: You have clicked variety pack.
+Action: click[0.8 ounce (pack of 24)]
+Observation: You have clicked 0.8 ounce (pack of 24).
+Action: click[Buy Now]
+Reflection: In this attempt, I was unsuccessful. I accidentally bought a product that was $100, which is more than my budget of $30. Either way, the initial search results were not good. Next time, I will do search[”variety pack of chips”] and then check if the results meet the dairy free and the $30 budget constraints. I will continue to refine my searches so that I can find more products.
+(input)
+Reflection:
+◄
+Feeling
+lucky?
+Conversion
+report
+Report
+an issue
+View original
+on arXiv
+►

research/notes/231004406-language-agent-tree-search-unifies-reasoning-acting-and-planning-in-la.md

@@ -0,0 +1,202 @@
+---
+title: '[2310.04406] Language Agent Tree Search Unifies Reasoning Acting and Planning
+  in Language Models'
+id: 231004406-language-agent-tree-search-unifies-reasoning-acting-and-planning-in-la
+tags:
+- deepread
+created: '2026-06-10T00:39:54.848871Z'
+source: https://arxiv.org/abs/2310.04406
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:39:54.848723Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2310.04406] Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
+Computer Science > Artificial Intelligence
+arXiv:2310.04406
+(cs)
+[Submitted on 6 Oct 2023 (
+v1
+), last revised 6 Jun 2024 (this version, v3)]
+Title:
+Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
+Authors:
+Andy Zhou
+,
+Kai Yan
+,
+Michal Shlapentokh-Rothman
+,
+Haohan Wang
+,
+Yu-Xiong Wang
+View a PDF of the paper titled Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models, by Andy Zhou and 4 other authors
+View PDF
+HTML (experimental)
+Abstract:
+While language models (LMs) have shown potential across a range of decision-making tasks, their reliance on simple acting processes limits their broad deployment as autonomous agents. In this paper, we introduce Language Agent Tree Search (LATS) -- the first general framework that synergizes the capabilities of LMs in reasoning, acting, and planning. By leveraging the in-context learning ability of LMs, we integrate Monte Carlo Tree Search into LATS to enable LMs as agents, along with LM-powered value functions and self-reflections for proficient exploration and enhanced decision-making. A key feature of our approach is the incorporation of an environment for external feedback, which offers a more deliberate and adaptive problem-solving mechanism that surpasses the constraints of existing techniques. Our experimental evaluation across diverse domains, including programming, interactive question-answering (QA), web navigation, and math, validates the effectiveness and generality of LATS in decision-making while maintaining competitive or improved reasoning performance. Notably, LATS achieves state-of-the-art pass@1 accuracy (92.7%) for programming on HumanEval with GPT-4 and demonstrates gradient-free performance (average score of 75.9) comparable to gradient-based fine-tuning for web navigation on WebShop with GPT-3.5. Code can be found at
+this https URL
+Comments:
+Code at
+this https URL
+Subjects:
+Artificial Intelligence (cs.AI)
+; Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
+Cite as:
+arXiv:2310.04406
+[cs.AI]
+(or
+arXiv:2310.04406v3
+[cs.AI]
+for this version)
+https://doi.org/10.48550/arXiv.2310.04406
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Andy Zhou [
+view email
+]
+[v1]
+Fri, 6 Oct 2023 17:55:11 UTC (371 KB)
+[v2]
+Tue, 5 Dec 2023 05:25:55 UTC (465 KB)
+[v3]
+Thu, 6 Jun 2024 02:51:17 UTC (960 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models, by Andy Zhou and 4 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.AI
+< prev
+|
+next >
+new
+|
+recent
+|
+2023-10
+Change to browse by:
+cs
+cs.CL
+cs.CV
+cs.LG
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/231006770-swe-bench-can-language-models-resolve-real-world-github-issues.md

@@ -0,0 +1,203 @@
+---
+title: '[2310.06770] SWE-bench: Can Language Models Resolve Real-World GitHub Issues?'
+id: 231006770-swe-bench-can-language-models-resolve-real-world-github-issues
+tags:
+- deepread
+created: '2026-06-10T00:23:35.577828Z'
+source: https://arxiv.org/abs/2310.06770
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:23:35.577638Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2310.06770] SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
+Computer Science > Computation and Language
+arXiv:2310.06770
+(cs)
+[Submitted on 10 Oct 2023 (
+v1
+), last revised 11 Nov 2024 (this version, v3)]
+Title:
+SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
+Authors:
+Carlos E. Jimenez
+,
+John Yang
+,
+Alexander Wettig
+,
+Shunyu Yao
+,
+Kexin Pei
+,
+Ofir Press
+,
+Karthik Narasimhan
+View a PDF of the paper titled SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, by Carlos E. Jimenez and 6 other authors
+View PDF
+Abstract:
+Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.
+Comments:
+Data, code, and leaderboard are available at
+this https URL
+ICLR 2024,
+this https URL
+Subjects:
+Computation and Language (cs.CL)
+; Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
+Cite as:
+arXiv:2310.06770
+[cs.CL]
+(or
+arXiv:2310.06770v3
+[cs.CL]
+for this version)
+https://doi.org/10.48550/arXiv.2310.06770
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Carlos E. Jimenez [
+view email
+]
+[v1]
+Tue, 10 Oct 2023 16:47:29 UTC (2,003 KB)
+[v2]
+Fri, 5 Apr 2024 18:16:29 UTC (2,258 KB)
+[v3]
+Mon, 11 Nov 2024 23:05:04 UTC (2,398 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, by Carlos E. Jimenez and 6 other authors
+View PDF
+TeX Source
+view license
+Current browse context:
+cs.CL
+< prev
+|
+next >
+new
+|
+recent
+|
+2023-10
+Change to browse by:
+cs
+cs.AI
+cs.SE
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/231108105-diloco-distributed-low-communication-training-of-language-models.md

@@ -0,0 +1,208 @@
+---
+title: '[2311.08105] DiLoCo: Distributed Low-Communication Training of Language Models'
+id: 231108105-diloco-distributed-low-communication-training-of-language-models
+tags:
+- deepread
+created: '2026-06-10T00:30:20.411067Z'
+source: https://arxiv.org/abs/2311.08105
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:30:20.410923Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2311.08105] DiLoCo: Distributed Low-Communication Training of Language Models
+Computer Science > Machine Learning
+arXiv:2311.08105
+(cs)
+[Submitted on 14 Nov 2023 (
+v1
+), last revised 23 Sep 2024 (this version, v3)]
+Title:
+DiLoCo: Distributed Low-Communication Training of Language Models
+Authors:
+Arthur Douillard
+,
+Qixuan Feng
+,
+Andrei A. Rusu
+,
+Rachita Chhaparia
+,
+Yani Donchev
+,
+Adhiguna Kuncoro
+,
+Marc'Aurelio Ranzato
+,
+Arthur Szlam
+,
+Jiajun Shen
+View a PDF of the paper titled DiLoCo: Distributed Low-Communication Training of Language Models, by Arthur Douillard and 8 other authors
+View PDF
+HTML (experimental)
+Abstract:
+Large language models (LLM) have become a critical component in many applications of machine learning. However, standard approaches to training LLM require a large number of tightly interconnected accelerators, with devices exchanging gradients and other intermediate states at each optimization step. While it is difficult to build and maintain a single computing cluster hosting many accelerators, it might be easier to find several computing clusters each hosting a smaller number of devices. In this work, we propose a distributed optimization algorithm, Distributed Low-Communication (DiLoCo), that enables training of language models on islands of devices that are poorly connected. The approach is a variant of federated averaging, where the number of inner steps is large, the inner optimizer is AdamW, and the outer optimizer is Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8 workers performs as well as fully synchronous optimization while communicating 500 times less. DiLoCo exhibits great robustness to the data distribution of each worker. It is also robust to resources becoming unavailable over time, and vice versa, it can seamlessly leverage resources that become available during training.
+Subjects:
+Machine Learning (cs.LG)
+; Computation and Language (cs.CL)
+Cite as:
+arXiv:2311.08105
+[cs.LG]
+(or
+arXiv:2311.08105v3
+[cs.LG]
+for this version)
+https://doi.org/10.48550/arXiv.2311.08105
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Arthur Douillard [
+view email
+]
+[v1]
+Tue, 14 Nov 2023 12:05:45 UTC (1,609 KB)
+[v2]
+Sat, 2 Dec 2023 14:10:14 UTC (1,610 KB)
+[v3]
+Mon, 23 Sep 2024 10:41:27 UTC (1,610 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled DiLoCo: Distributed Low-Communication Training of Language Models, by Arthur Douillard and 8 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.LG
+< prev
+|
+next >
+new
+|
+recent
+|
+2023-11
+Change to browse by:
+cs
+cs.CL
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+IArxiv recommender toggle
+IArxiv Recommender
+(
+What is IArxiv?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/231108516-llms-cannot-find-reasoning-errors-but-can-correct-them-given-the-error.md

@@ -0,0 +1,204 @@
+---
+title: '[2311.08516] LLMs cannot find reasoning errors, but can correct them given
+  the error location'
+id: 231108516-llms-cannot-find-reasoning-errors-but-can-correct-them-given-the-error
+tags:
+- deepread
+created: '2026-06-10T00:40:16.980357Z'
+source: https://arxiv.org/abs/2311.08516
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:40:16.980220Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2311.08516] LLMs cannot find reasoning errors, but can correct them given the error location
+Computer Science > Artificial Intelligence
+arXiv:2311.08516
+(cs)
+[Submitted on 14 Nov 2023 (
+v1
+), last revised 4 Jun 2024 (this version, v3)]
+Title:
+LLMs cannot find reasoning errors, but can correct them given the error location
+Authors:
+Gladys Tyen
+,
+Hassan Mansoor
+,
+Victor Cărbune
+,
+Peter Chen
+,
+Tony Mak
+View a PDF of the paper titled LLMs cannot find reasoning errors, but can correct them given the error location, by Gladys Tyen and 4 other authors
+View PDF
+HTML (experimental)
+Abstract:
+While self-correction has shown promise in improving LLM outputs in terms of style and quality (e.g. Chen et al., 2023b; Madaan et al., 2023), recent attempts to self-correct logical or reasoning errors often cause correct answers to become incorrect, resulting in worse performances overall (Huang et al., 2023). In this paper, we show that poor self-correction performance stems from LLMs' inability to find logical mistakes, rather than their ability to correct a known mistake. Firstly, we benchmark several state-of-the-art LLMs on their mistake-finding ability and demonstrate that they generally struggle with the task, even in highly objective, unambiguous cases. Secondly, we test the correction abilities of LLMs -- separately from mistake finding -- using a backtracking setup that feeds ground truth mistake location information to the model. We show that this boosts downstream task performance across our 5 reasoning tasks, indicating that LLMs' correction abilities are robust. Finally, we show that it is possible to obtain mistake location information without ground truth labels or in-domain training data. We train a small classifier with out-of-domain data, which exhibits stronger mistake-finding performance than prompting a large model. We release our dataset of LLM-generated logical mistakes, BIG-Bench Mistake, to enable further research into locating LLM reasoning mistakes.
+Comments:
+ACL 2024 Findings
+Subjects:
+Artificial Intelligence (cs.AI)
+; Computation and Language (cs.CL); Machine Learning (cs.LG)
+Cite as:
+arXiv:2311.08516
+[cs.AI]
+(or
+arXiv:2311.08516v3
+[cs.AI]
+for this version)
+https://doi.org/10.48550/arXiv.2311.08516
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Gladys Tyen [
+view email
+]
+[v1]
+Tue, 14 Nov 2023 20:12:38 UTC (7,191 KB)
+[v2]
+Tue, 9 Jan 2024 03:32:32 UTC (7,191 KB)
+[v3]
+Tue, 4 Jun 2024 10:25:13 UTC (7,319 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled LLMs cannot find reasoning errors, but can correct them given the error location, by Gladys Tyen and 4 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.AI
+< prev
+|
+next >
+new
+|
+recent
+|
+2023-11
+Change to browse by:
+cs
+cs.CL
+cs.LG
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+Links to Code Toggle
+Papers with Code
+(
+What is Papers with Code?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/231209152-evaluating-augmented-reality-communication-how-can-we-teach-procedural.md

@@ -0,0 +1,203 @@
+---
+title: '[2312.09152] Evaluating Augmented Reality Communication: How Can We Teach
+  Procedural Skill in AR?'
+id: 231209152-evaluating-augmented-reality-communication-how-can-we-teach-procedural
+tags:
+- deepread
+created: '2026-06-10T00:40:10.692238Z'
+source: https://arxiv.org/abs/2312.09152
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:40:10.692096Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2312.09152] Evaluating Augmented Reality Communication: How Can We Teach Procedural Skill in AR?
+Computer Science > Human-Computer Interaction
+arXiv:2312.09152
+(cs)
+[Submitted on 14 Dec 2023]
+Title:
+Evaluating Augmented Reality Communication: How Can We Teach Procedural Skill in AR?
+Authors:
+Manuel Rebol
+,
+Krzysztof Pietroszek
+,
+Neal Sikka
+,
+Claudia Ranniger
+,
+Colton Hood
+,
+Adam Rutenberg
+,
+Puja Sasankan
+,
+Christian Gütl
+View a PDF of the paper titled Evaluating Augmented Reality Communication: How Can We Teach Procedural Skill in AR?, by Manuel Rebol and 7 other authors
+View PDF
+HTML (experimental)
+Abstract:
+Augmented reality (AR) has great potential for use in healthcare applications, especially remote medical training and supervision. In this paper, we analyze the usage of an AR communication system to teach a medical procedure, the placement of a central venous catheter (CVC) under ultrasound guidance. We examine various AR communication and collaboration components, including gestural communication, volumetric information, annotations, augmented objects, and augmented screens. We compare how teaching in AR differs from teaching through videoconferencing-based communication. Our results include a detailed medical training steps analysis in which we compare how verbal and visual communication differs between video and AR training. We identify procedural steps in which medical experts give visual instructions utilizing AR components. We examine the change in AR usage and interaction over time and recognize patterns between users. Moreover, AR design recommendations are given based on post-training interviews.
+Comments:
+this https URL
+Subjects:
+Human-Computer Interaction (cs.HC)
+Cite as:
+arXiv:2312.09152
+[cs.HC]
+(or
+arXiv:2312.09152v1
+[cs.HC]
+for this version)
+https://doi.org/10.48550/arXiv.2312.09152
+Focus to learn more
+arXiv-issued DOI via DataCite
+Journal reference:
+Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology (VRST 2023)
+Related DOI
+:
+https://doi.org/10.1145/3611659.3615685
+Focus to learn more
+DOI(s) linking to related resources
+Submission history
+From: Manuel Rebol [
+view email
+]
+[v1]
+Thu, 14 Dec 2023 17:22:22 UTC (2,671 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Evaluating Augmented Reality Communication: How Can We Teach Procedural Skill in AR?, by Manuel Rebol and 7 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.HC
+< prev
+|
+next >
+new
+|
+recent
+|
+2023-12
+Change to browse by:
+cs
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/240201817-llms-cant-plan-but-can-help-planning-in-llm-modulo-frameworks.md

@@ -0,0 +1,203 @@
+---
+title: '[2402.01817] LLMs Can''t Plan, But Can Help Planning in LLM-Modulo Frameworks'
+id: 240201817-llms-cant-plan-but-can-help-planning-in-llm-modulo-frameworks
+tags:
+- deepread
+created: '2026-06-10T00:40:16.134883Z'
+source: https://arxiv.org/abs/2402.01817
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:40:16.134739Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2402.01817] LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
+Computer Science > Artificial Intelligence
+arXiv:2402.01817
+(cs)
+[Submitted on 2 Feb 2024 (
+v1
+), last revised 12 Jun 2024 (this version, v3)]
+Title:
+LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
+Authors:
+Subbarao Kambhampati
+,
+Karthik Valmeekam
+,
+Lin Guan
+,
+Mudit Verma
+,
+Kaya Stechly
+,
+Siddhant Bhambri
+,
+Lucas Saldyt
+,
+Anil Murthy
+View a PDF of the paper titled LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks, by Subbarao Kambhampati and 7 other authors
+View PDF
+HTML (experimental)
+Abstract:
+There is considerable confusion about the role of Large Language Models (LLMs) in planning and reasoning tasks. On one side are over-optimistic claims that LLMs can indeed do these tasks with just the right prompting or self-verification strategies. On the other side are perhaps over-pessimistic claims that all that LLMs are good for in planning/reasoning tasks are as mere translators of the problem specification from one syntactic format to another, and ship the problem off to external symbolic solvers. In this position paper, we take the view that both these extremes are misguided. We argue that auto-regressive LLMs cannot, by themselves, do planning or self-verification (which is after all a form of reasoning), and shed some light on the reasons for misunderstandings in the literature. We will also argue that LLMs should be viewed as universal approximate knowledge sources that have much more meaningful roles to play in planning/reasoning tasks beyond simple front-end/back-end format translators. We present a vision of {\bf LLM-Modulo Frameworks} that combine the strengths of LLMs with external model-based verifiers in a tighter bi-directional interaction regime. We will show how the models driving the external verifiers themselves can be acquired with the help of LLMs. We will also argue that rather than simply pipelining LLMs and symbolic components, this LLM-Modulo Framework provides a better neuro-symbolic approach that offers tighter integration between LLMs and symbolic components, and allows extending the scope of model-based planning/reasoning regimes towards more flexible knowledge, problem and preference specifications.
+Subjects:
+Artificial Intelligence (cs.AI)
+; Machine Learning (cs.LG)
+Cite as:
+arXiv:2402.01817
+[cs.AI]
+(or
+arXiv:2402.01817v3
+[cs.AI]
+for this version)
+https://doi.org/10.48550/arXiv.2402.01817
+Focus to learn more
+arXiv-issued DOI via DataCite
+Journal reference:
+Proceedings of the 41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024
+Submission history
+From: Subbarao Kambhampati [
+view email
+]
+[v1]
+Fri, 2 Feb 2024 14:43:18 UTC (4,551 KB)
+[v2]
+Tue, 6 Feb 2024 01:29:37 UTC (4,552 KB)
+[v3]
+Wed, 12 Jun 2024 01:13:11 UTC (6,405 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks, by Subbarao Kambhampati and 7 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.AI
+< prev
+|
+next >
+new
+|
+recent
+|
+2024-02
+Change to browse by:
+cs
+cs.LG
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/240203300-deepseekmath-pushing-the-limits-of-mathematical-reasoning-in-open-lang.md

@@ -0,0 +1,214 @@
+---
+title: '[2402.03300] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in
+  Open Language Models'
+id: 240203300-deepseekmath-pushing-the-limits-of-mathematical-reasoning-in-open-lang
+tags:
+- deepread
+created: '2026-06-09T23:28:29.232007Z'
+source: https://arxiv.org/abs/2402.03300
+source_domain: arxiv.org
+fetched_at: '2026-06-09T23:28:29.231807Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2402.03300] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
+Computer Science > Computation and Language
+arXiv:2402.03300
+(cs)
+[Submitted on 5 Feb 2024 (
+v1
+), last revised 27 Apr 2024 (this version, v3)]
+Title:
+DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
+Authors:
+Zhihong Shao
+,
+Peiyi Wang
+,
+Qihao Zhu
+,
+Runxin Xu
+,
+Junxiao Song
+,
+Xiao Bi
+,
+Haowei Zhang
+,
+Mingchuan Zhang
+,
+Y.K. Li
+,
+Y. Wu
+,
+Daya Guo
+View a PDF of the paper titled DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, by Zhihong Shao and 10 other authors
+View PDF
+HTML (experimental)
+Abstract:
+Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.
+Subjects:
+Computation and Language (cs.CL)
+; Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
+Cite as:
+arXiv:2402.03300
+[cs.CL]
+(or
+arXiv:2402.03300v3
+[cs.CL]
+for this version)
+https://doi.org/10.48550/arXiv.2402.03300
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Zhihong Shao [
+view email
+]
+[v1]
+Mon, 5 Feb 2024 18:55:32 UTC (3,417 KB)
+[v2]
+Tue, 6 Feb 2024 18:39:38 UTC (3,417 KB)
+[v3]
+Sat, 27 Apr 2024 15:25:53 UTC (3,417 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, by Zhihong Shao and 10 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.CL
+< prev
+|
+next >
+new
+|
+recent
+|
+2024-02
+Change to browse by:
+cs
+cs.AI
+cs.LG
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+Links to Code Toggle
+Papers with Code
+(
+What is Papers with Code?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/240411018-many-shot-in-context-learning.md

@@ -0,0 +1,228 @@
+---
+title: '[2404.11018] Many-Shot In-Context Learning'
+id: 240411018-many-shot-in-context-learning
+tags:
+- deepread
+created: '2026-06-10T00:40:15.011649Z'
+source: https://arxiv.org/abs/2404.11018
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:40:15.011513Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2404.11018] Many-Shot In-Context Learning
+Computer Science > Machine Learning
+arXiv:2404.11018
+(cs)
+[Submitted on 17 Apr 2024 (
+v1
+), last revised 17 Oct 2024 (this version, v3)]
+Title:
+Many-Shot In-Context Learning
+Authors:
+Rishabh Agarwal
+,
+Avi Singh
+,
+Lei M. Zhang
+,
+Bernd Bohnet
+,
+Luis Rosias
+,
+Stephanie Chan
+,
+Biao Zhang
+,
+Ankesh Anand
+,
+Zaheer Abbas
+,
+Azade Nova
+,
+John D. Co-Reyes
+,
+Eric Chu
+,
+Feryal Behbahani
+,
+Aleksandra Faust
+,
+Hugo Larochelle
+View a PDF of the paper titled Many-Shot In-Context Learning, by Rishabh Agarwal and 13 other authors
+View PDF
+HTML (experimental)
+Abstract:
+Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, without any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to fine-tuning. We also find that inference cost increases linearly in the many-shot regime, and frontier LLMs benefit from many-shot ICL to varying degrees. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.
+Comments:
+NeurIPS (Spotlight)
+Subjects:
+Machine Learning (cs.LG)
+; Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
+Cite as:
+arXiv:2404.11018
+[cs.LG]
+(or
+arXiv:2404.11018v3
+[cs.LG]
+for this version)
+https://doi.org/10.48550/arXiv.2404.11018
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Rishabh Agarwal [
+view email
+]
+[v1]
+Wed, 17 Apr 2024 02:49:26 UTC (327 KB)
+[v2]
+Wed, 22 May 2024 17:06:10 UTC (370 KB)
+[v3]
+Thu, 17 Oct 2024 17:45:09 UTC (414 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Many-Shot In-Context Learning, by Rishabh Agarwal and 13 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.LG
+< prev
+|
+next >
+new
+|
+recent
+|
+2024-04
+Change to browse by:
+cs
+cs.AI
+cs.CL
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+Links to Code Toggle
+Papers with Code
+(
+What is Papers with Code?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+IArxiv recommender toggle
+IArxiv Recommender
+(
+What is IArxiv?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/240612543-phase-controlled-heat-modulation-with-aharonov-bohm-interferometers.md

@@ -0,0 +1,197 @@
+---
+title: '[2406.12543] Phase-controlled heat modulation with Aharonov-Bohm interferometers'
+id: 240612543-phase-controlled-heat-modulation-with-aharonov-bohm-interferometers
+tags:
+- deepread
+created: '2026-06-10T00:40:09.876451Z'
+source: https://arxiv.org/abs/2406.12543
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:40:09.876309Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2406.12543] Phase-controlled heat modulation with Aharonov-Bohm interferometers
+Condensed Matter > Mesoscale and Nanoscale Physics
+arXiv:2406.12543
+(cond-mat)
+[Submitted on 18 Jun 2024]
+Title:
+Phase-controlled heat modulation with Aharonov-Bohm interferometers
+Authors:
+Sun-Yong Hwang
+,
+Björn Sothmann
+,
+Rosa López
+View a PDF of the paper titled Phase-controlled heat modulation with Aharonov-Bohm interferometers, by Sun-Yong Hwang and 2 other authors
+View PDF
+HTML (experimental)
+Abstract:
+A heat modulator is proposed based on a voltage-biased Aharonov-Bohm interferometer. Once an electrical bias is applied, Peltier effects give rise to a flow of heat that can be modulated by a magnetic flux. We determine the corresponding temperature changes using a simple thermal model. Our calculations demonstrate that the modulated temperature difference can be as large as 80 mK at base temperature about 600 mK with relative temperature variations reaching 10\%. Our model also predicts, quite generally, the emergence of spin-polarized heat flows without any ferromagnetic contacts, if Rashba spin-orbit interaction is combined with the applied magnetic flux, which potentially paves the way towards caloritronic information processing.
+Comments:
+8 pages, 4 figures
+Subjects:
+Mesoscale and Nanoscale Physics (cond-mat.mes-hall)
+Cite as:
+arXiv:2406.12543
+[cond-mat.mes-hall]
+(or
+arXiv:2406.12543v1
+[cond-mat.mes-hall]
+for this version)
+https://doi.org/10.48550/arXiv.2406.12543
+Focus to learn more
+arXiv-issued DOI via DataCite
+Journal reference:
+Phys. Rev. Research 6, 013215 (2024)
+Related DOI
+:
+https://doi.org/10.1103/PhysRevResearch.6.013215
+Focus to learn more
+DOI(s) linking to related resources
+Submission history
+From: Sun-Yong Hwang [
+view email
+]
+[v1]
+Tue, 18 Jun 2024 12:22:44 UTC (1,894 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Phase-controlled heat modulation with Aharonov-Bohm interferometers, by Sun-Yong Hwang and 2 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cond-mat.mes-hall
+< prev
+|
+next >
+new
+|
+recent
+|
+2024-06
+Change to browse by:
+cond-mat
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+IArxiv recommender toggle
+IArxiv Recommender
+(
+What is IArxiv?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/240806195-mutual-reasoning-makes-smaller-llms-stronger-problem-solvers-2.md

@@ -0,0 +1,2384 @@
+---
+title: '[2408.06195] Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers'
+id: 240806195-mutual-reasoning-makes-smaller-llms-stronger-problem-solvers-2
+tags:
+- deepread
+created: '2026-06-10T00:40:45.751799Z'
+source: https://ar5iv.labs.arxiv.org/html/2408.06195
+source_domain: ar5iv.labs.arxiv.org
+fetched_at: '2026-06-10T00:40:45.751621Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2408.06195] Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers
+Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers
+Zhenting Qi
+∗‡†
+Mingyuan Ma
+∗‡†
+Jiahang Xu
+∗‡
+Li Lyna Zhang
+‡⋄
+Fan Yang
+‡
+Mao Yang
+‡
+‡
+Microsoft Research Asia
+†
+Harvard University
+Abstract
+This paper introduces rStar, a self-play mutual reasoning approach that significantly improves reasoning capabilities of small language models (SLMs) without fine-tuning or superior models. rStar decouples reasoning into a self-play mutual generation-discrimination process. First, a target SLM augments the Monte Carlo Tree Search (MCTS) with
+a rich set of human-like reasoning actions
+to construct higher quality reasoning trajectories. Next, another SLM, with capabilities similar to the target SLM, acts as a discriminator to verify each trajectory generated by the target SLM. The mutually agreed reasoning trajectories are considered
+mutual consistent
+, thus are more likely to be correct. Extensive experiments across five SLMs demonstrate rStar can effectively solve diverse reasoning problems, including GSM8K, GSM-Hard, MATH, SVAMP, and StrategyQA. Remarkably, rStar boosts GSM8K accuracy from 12.51% to 63.91% for LLaMA2-7B, from 36.46% to 81.88% for Mistral-7B, from 74.53% to 91.13% for LLaMA3-8B-Instruct. Code will be available at
+here
+.
+$*$
+$*$
+footnotetext:
+Equal contribution. Zhenting Qi and Mingyuan Ma did the work during an internship at MSRA
+$\diamond$
+$\diamond$
+footnotetext:
+Corresponding author: lzhani@microsoft.com
+1
+Introduction
+Despite their success, large language models (LLMs) face significant challenges in complex reasoning
+(Valmeekam et al.,
+2022
+; Weng et al.,
+2023
+)
+. For example, state of the art models like Mistral-7B
+(Jiang et al.,
+2023
+)
+can only achieve 36.5% accuracy on the GSM8K dataset, even with techniques like Chain-of-Throught (CoT)
+(Wei et al.,
+2022
+)
+. Although fine-tuning is shown to be an effective way to improve reasoning capability, most LLMs rely on fine-tuning data distilled or synthesized by
+superior
+models like GPT-4
+(Wang et al.,
+2024a
+; Gou et al.,
+2023
+)
+. Meanwhile, the community has been actively working on a complimentary and yet more challenging approach: Reasoning improvements
+without
+a superior teacher LLM.
+Figure 1:
+With 32 rounds of inference, rStar makes SLMs highly capable problem-solvers, matching or even surpassing the reasoning performance achieved after domain-specialized SFT.
+A promising paradigm to improve reasoning without superior models is to leverage the knowledge within LLMs themselves
+(Wang et al.,
+2023
+; Hao et al.,
+2023
+; Madaan et al.,
+2024
+)
+. For example, RAP
+(Hao et al.,
+2023
+)
+adopts a self-exploration solution to iteratively improve LLM’s reasoning performance through self-rewarded feedback. Unfortunately, study suggests that this paradigm often suffers from two fundamental issues.
+First, LLMs often struggle to effectively explore the solution space during reasoning. The self-exploration often traps in a solution space with low-quality reasoning steps even after many attempts. For example, our experiments reveal that after 32 rounds of self-exploration with RAP
+(Hao et al.,
+2023
+)
+, only 24% of the trajectories generated by LLaMA2-7B on GSM8K are correct.
+Second, even the self-exploration can find high quality reasoning steps, it is difficult for SLMs to tell which reasoning steps are of higher quality or determine which final answers are correct, thus it is hard to effectively guide the self-exploration. Our study shows that a naïve reward-based self-exploration guidance can lead to results no better than random guesses (see Appendix
+A.1
+).
+A more troublesome fact is that the above two issues are more pronounced in the smaller version of LLMs, i.e.,
+SLM
+s, due to their weaker capabilities. For instance, while GPT-4 can improve by self-refining its output
+(Madaan et al.,
+2024
+; Wu et al.,
+2024
+; Zhou et al.,
+2024
+)
+, the approaches are less effective in SLMs and may even lead to worse performance
+(Forsman,
+2024
+)
+. This significantly hinders the adoption of neural language models.
+This paper introduces
+S
+elf-play mu
+T
+u
+A
+l
+R
+easoning
+(rStar), a novel approach that boosts SLMs’ reasoning capability during inference without fine-tuning or superior models. To address the aforementioned challenges, rStar decouples reasoning into a self-play mutual generation-discrimination process as illustrated in Fig.
+2
+.
+Specifically, rStar is unique in the following approaches. First, although relying on a conventional Monte Carlo Tree Search (MCTS) for SLMs to self-generate reasoning steps, rStar advocates
+a richer set of reasoning actions
+in the self-exploration. The new proposed actions simulate human reasoning behaviors given the current reasoning state, such as decomposing and searching for a specific reasoning step, proposing a new sub-question, or rephrasing the given question. This enables SLMs to generate high-quality candidate reasoning trajectories during self-exploration.
+Second, to effectively guide the exploration among the generated reasoning trajectories, rStar augments the MCTS process with a new discrimination process called
+mutual consistency
+. In particular, rStar employs a second SLM with the similar capability, acting as a discriminator to provide unsupervised feedback on each candidate reasoning trajectory generated by MCTS. To improve the accuracy of the feedback, rStar hints the second SLM with sampled partial reasoning trajectories, asking it to complete the remaining reasoning steps. And rStar deems the mutually agreed reasoning trajectories of higher quality. Mutual consistency mirrors the common human practice in the absence of supervision, where agreement among peers (i.e., two SLMs) on derived answers suggests a higher likelihood of correctness.
+As a result, mutual consistency offers more effective reasoning across diverse tasks than other approaches like self-consistency
+(Wang et al.,
+2023
+)
+and avoids the risk of overfitting when training a reward model
+(Chen et al.,
+2024a
+; Wang et al.,
+2024b
+)
+.
+Figure 2:
+Our self-play mutual reasoning is a generation-discrimination process: (1) a self-generator augments the target SLM to generate candidate reasoning trajectories using MCTS; (2) the discriminator uses another SLM to provide unsupervised feedback on each trajectory based on partial hints; (3) based on this feedback, the target SLM decides a final reasoning trajectory as the solution.
+Extensive experiments across five SLMs and five diverse reasoning tasks demonstrate the effectiveness of rStar. With just 32 rounds of MCTS inference, rStar significantly enhances SLMs’ reasoning capabilities, matching or even surpassing the accuracy achieved after fine-tuning. For example, rStar boosts GSM8K accuracy from 12.51% to 63.91% for LLaMA2-7B, from 36.46% to 81.88% for Mistral, and from 47.23% to 85.52% for LLaMA3-8B. Furthermore, we conduct comprehensive experiments to verify rStar’s superiority over state-of-the-art baselines, including single-round inference techniques like few-shot CoT, multi-round prompting approaches such as self-consistency, and self-improvement techniques such as RAP, ToT, self-evaluation and self-verification.
+2
+Related Work
+Prompting Language Models to Reason
+.
+Prompting-based methods, such as Chain-of-Thought
+(Wei et al.,
+2022
+)
+, focus on designing instructions and pipelines to enhance LLMs’ reasoning performance during inference. Recent advances include planning
+(Hao et al.,
+2023
+; Ding et al.,
+2023
+)
+, problem decomposition
+(Zhou et al.,
+2022
+; Khot et al.,
+2022
+; Hao et al.,
+2023
+)
+, abstraction
+(Zheng et al.,
+2023
+)
+, programming
+(Chen et al.,
+2022
+; Zhou et al.,
+2023
+)
+.
+These methods aim to improve single-round inference performance and are orthogonal to ours.
+LLM Self-improvement
+. Recently, research on the self-improvement of LLMs has rapidly increased.
+Fine-tuning based methods
+(Chen et al.,
+2024b
+;
+a
+)
+leverage the capabilities of a well-pretrained LLM to synthesize data and progressively enhance its performance. Advanced prompting techniques, such as self-verification
+(Gero et al.,
+2023
+; Zhou et al.,
+2023
+)
+, and RAP
+(Hao et al.,
+2023
+)
+, improve performance through iterative self-exploring based on self-diagnosed feedback at inference time.
+However, as illustrated in previous section, the achieved performance often depend on the LLM’s inherent capabilities, and for SLMs, their weaker instruction-following ability and unreliable self-rewarding can mislead self-improvement.
+Sampling Reasoning Paths
+. Recent works
+(Brown et al.,
+2024
+; Li et al.,
+2024
+; Snell et al.,
+2024
+)
+on mathematical reasoning have shown that sampling diverse reasoning paths can significantly enhance performance compared to greedy one-time decoding. Self-Consistency
+(Wang et al.,
+2023
+)
+sample a complete CoT path each time. Tree-search approaches
+(Yao et al.,
+2024
+; Hao et al.,
+2023
+; Zhang et al.,
+2024
+)
+, like MCTS, further improve the performance by breaking down tasks and sampling simpler, individual intermediate reasoning steps. However, most approaches have limited action spaces. For example, RAP
+(Hao et al.,
+2023
+)
+decomposes only subproblems, while AlphaMath
+(Chen et al.,
+2024a
+)
+searches only for one CoT step, limiting effectiveness in generating better trajectories.
+Answer Verification
+. To select correct reasoning trajectories, majority voting
+(Wang et al.,
+2023
+)
+is a widely-used approach. To improve accuracy, some works train value or rewards model for verification
+(Wang et al.,
+2024b
+; Chen et al.,
+2024a
+)
+, but these require additional annotations and have risks in overfitting to specific tasks. Self-verification
+(Weng et al.,
+2023
+)
+leverages LLM capabilities for backward self-verification. Nevertheless, its effectiveness hinges on its inherent ability to reason effectively. Recent studies have shown that LLM struggles to evaluate itself and rectify its initial responses without any external feedbacks
+(Huang et al.,
+2023
+; Feng et al.,
+2023
+)
+.
+3
+Methodology
+3.1
+Overview
+Problem Formulation
+. To solve a reasoning problem by SLMs, we formulate it as a multi-step reasoning generation task, which breaks
+the problem into simpler sub-tasks. This is more effective than traditional CoT-based reasoning
+(Wei et al.,
+2022
+; Wang et al.,
+2023
+)
+, as it is much easier for SLMs to correctly generate one step than complete reasoning steps in a single inference. We leverage the Monte-Carlo Tree Search (MCTS) algorithm
+(Kocsis & Szepesvári,
+2006
+)
+to augment the target SLM for self-generating multi-step reasoning solutions.
+Formally, for a given problem
+x
+𝑥
+x
+and a target SLM
+M
+𝑀
+M
+, the MCTS augments
+M
+𝑀
+M
+to incrementally build a search tree
+𝒯
+𝒯
+\mathcal{T}
+. As illustrated in Fig.
+3
+, the root node represents the question
+x
+𝑥
+x
+, an edge represents an action
+a
+𝑎
+a
+, each child node is an intermediate step
+s
+𝑠
+s
+generated by
+M
+𝑀
+M
+under the corresponding action. A path from the root node to a leaf node (denoted as
+s
+d
+subscript
+𝑠
+𝑑
+s_{d}
+, also called a terminal node) constitutes a candidate solution trajectory
+𝐭
+=
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+d
+𝐭
+direct-sum
+𝑥
+subscript
+𝑠
+1
+subscript
+𝑠
+2
+…
+subscript
+𝑠
+𝑑
+\mathbf{t}=x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{d}
+. From the search tree
+𝒯
+𝒯
+\mathcal{T}
+, we can extract a set of solution trajectories
+𝕋
+=
+{
+𝐭
+1
+,
+𝐭
+2
+,
+…
+,
+𝐭
+n
+}
+​
+(
+n
+≥
+1
+)
+𝕋
+superscript
+𝐭
+1
+superscript
+𝐭
+2
+…
+superscript
+𝐭
+𝑛
+𝑛
+1
+\mathbb{T}=\{\mathbf{t}^{1},\mathbf{t}^{2},...,\mathbf{t}^{n}\}(n\geq 1)
+. Our goal is to find the trajectories that can achieve the correct answer for the given question.
+Challenges in SLM Self-Improvement
+. MCTS allows an SLM to explore and evaluate multiple potential solutions. Ideally, by balancing exploration of new possibilities with the exploitation of high-reward actions, the SLM can gradually refine its reasoning steps to generate a final correct reasoning trajectory. However, due to the limited capabilities in SLMs, traditional MCTS yields minimal improvement. First, the vast solution space makes it challenging for SLMs to generate effective solutions. Existing MCTS-based methods
+(Hao et al.,
+2023
+; Kang et al.,
+2024
+)
+that use single actions limit diversity and struggle to generalize across tasks. Approaches like self-consistency
+(Wang et al.,
+2023
+)
+use random sampling ensure diversity, SLMs often produce poor-quality solutions, requiring many attempts to find a correct solution, thereby increasing inference costs.
+Second, it’s challenging to accurately reward each action. Without ground truth labels, it’s difficult to verify the correctness for each intermediate step
+s
+i
+subscript
+𝑠
+𝑖
+s_{i}
+and the final answer in
+s
+d
+subscript
+𝑠
+𝑑
+s_{d}
+. Majority voting in self-consistency requires most traces to be correct, which is often not the case for SLMs. Methods like RAP
+(Hao et al.,
+2023
+)
+use self-rewarding, but our study shows SLMs perform near-random self-rewarding (Appendix
+A.1
+). Training a reward model, as in M
+∗
+(Kang et al.,
+2024
+)
+, can address this challenge but faces difficulties in collecting training data and generalizing across various tasks.
+Overview
+.
+To address these challenges, this section introduces our methodology, rStar, which decomposes reasoning into solution generation and mutual verification in Fig.
+2
+. To tackle the first challenge, we introduce a richer set of human-like reasoning actions that allows for thorough space exploration across diverse reasoning tasks. To address the second challenge, we design an SLM-tailored reward function to evaluate intermediate steps, avoiding reliance on their often unreliable self-evaluations. Moreover, we use another SLM as a discriminator to augment the MCTS process, mutually verifying the correctness of each trajectory with the generator SLM.
+3.2
+Self-generating Reasoning Trajectory with MCTS Rollout
+Figure 3:
+An example to illustrate the process of self-generator. Highlighted nodes from top to bottom constitute a complete reasoning trace.
+Given a question, MCTS augments the target SLM to explore a rich, human-like reasoning action space and generate the next steps based on the current state.
+A Rich Set of Human-like Reasoning Actions
+. At the core of MCTS generation lies the action space, which defines the scope of tree exploration. Most MCTS-based methods use a single action type to build the tree. For instance, in RAP, the action is to propose the next sub-question, whereas in AlphaMath
+(Chen et al.,
+2024a
+)
+and MindStar
+(Kang et al.,
+2024
+)
+, the action is to generate the next reasoning step.
+However, relying on a single action type can easily lead to ineffective space exploration.
+To address this, we revisit how humans approach reasoning.
+Different people solve problems using diverse actions: some break into sub-questions, others solve it directly, and some might rephrase the problem to focus on key conditions. Moreover, people adjust their approach based on current states, choosing different actions as needed. Inspired by this human reasoning process, we introduce a richer set of 5 actions to maximize the SLM’s potential for correctly solving complex reasoning problems.
+⋄
+⋄
+\diamond
+A1
+: Propose an one-step thought
+. This action prompts the LLM to generate the next one-step thought for a given question, by considering the existing reasoning steps. Unlike the CoT, which generates complete thoughts, this approach simplifies the reasoning process and allows the LLM to perform better decision making
+(Yao et al.,
+2024
+; Besta et al.,
+2024
+)
+.
+⋄
+⋄
+\diamond
+A2
+: Propose the remaining thought steps.
+Instead of generating only one step thought per state, this action aligns with standard CoT, enabling “fast thinking” to solve simple question in fewer steps. Given the already generated reasoning steps, it prompts the LLM to directly produce the remaining steps until reaching the final answer.
+⋄
+⋄
+\diamond
+A3
+: Propose next sub-question along with its answer.
+This action is inspired by
+least-to-most prompting
+(Zhou et al.,
+2022
+)
+, which breaks down a complex problem into a series of simpler sub-questions and solves them sequentially. Following RAP’s implementation, we prompt the LLM to ask and then answer the next sub-question.
+⋄
+⋄
+\diamond
+A4
+: Answer the sub-question again.
+Considering that a sub-question might not be answered correctly by
+A3
+, we propose this action to re-answer it. To improve accuracy, this action prompts the LLM to use few-shot CoT. Note that the original answer generated by
+A3
+did not use a CoT-like prompt but instead followed the least-to-most problem decomposition prompt
+(Zhou et al.,
+2022
+)
+.
+⋄
+⋄
+\diamond
+A5
+: Rephrase the question/sub-question.
+When analyzing incorrect cases, we found that many of them are due the LLM misunderstanding the question. For example, it might miss a specific condition provided in the question. Therefore, we propose a new action to rephrase the question more simply. Specifically, we prompt the LLM to clearly list all conditions given in the problem statement.
+Table 1:
+Ablation study on the effectiveness of our rich action space: we evaluate LLaMA3-8B on 200 sampled GSM8K questions.
+Action Space
+Accuracy
+A
+3
+subscript
+𝐴
+3
+A_{3}
+(i.e., RAP)
+70.5
+A
+3
+subscript
+𝐴
+3
+A_{3}
++
+A
+5
+subscript
+𝐴
+5
+A_{5}
+72.5
+A
+3
+subscript
+𝐴
+3
+A_{3}
++
+A
+4
+subscript
+𝐴
+4
+A_{4}
++
+A
+5
+subscript
+𝐴
+5
+A_{5}
+73.5
+A
+2
+subscript
+𝐴
+2
+A_{2}
++
+A
+3
+subscript
+𝐴
+3
+A_{3}
++
+A
+4
+subscript
+𝐴
+4
+A_{4}
++
+A
+5
+subscript
+𝐴
+5
+A_{5}
+74.0
+All (
+A
+1
+subscript
+𝐴
+1
+A_{1}
++
+A
+2
+subscript
+𝐴
+2
+A_{2}
++
+A
+3
+subscript
+𝐴
+3
+A_{3}
++
+A
+4
+subscript
+𝐴
+4
+A_{4}
++
+A
+5
+subscript
+𝐴
+5
+A_{5}
+)
+75.0
+The above 5 actions define a highly diverse action space
+{
+A
+1
+,
+A
+2
+,
+A
+3
+,
+A
+4
+,
+A
+5
+}
+subscript
+𝐴
+1
+subscript
+𝐴
+2
+subscript
+𝐴
+3
+subscript
+𝐴
+4
+subscript
+𝐴
+5
+\{A_{1},A_{2},A_{3},A_{4},A_{5}\}
+.
+At each step
+i
+𝑖
+i
+, MCTS selects an action
+a
+i
+subscript
+𝑎
+𝑖
+a_{i}
+from this space. We then use this action
+a
+i
+subscript
+𝑎
+𝑖
+a_{i}
+to prompt the LLM to generate the next reasoning step
+s
+i
+subscript
+𝑠
+𝑖
+s_{i}
+, based on the current state, which is the previous generated trajectory
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+i
+−
+1
+direct-sum
+𝑥
+subscript
+𝑠
+1
+subscript
+𝑠
+2
+…
+subscript
+𝑠
+𝑖
+1
+x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{i-1}
+. Note that certain actions require orders. For example,
+A4
+can only happen after
+A3
+, and
+A5
+can only happen after the root question. As shown in Table
+1
+, each action plays a crucial role in improving the final reasoning accuracy.
+Reward Function.
+Another critical component in MCTS is the reward function, which evaluates the value of each action and directs the tree expansion.
+We design a simple yet effective reward function for SLMs. First, we exclude self-rewarding techniques for any intermediate nodes due to the limited capabilities of SLMs. Second, to ensure generalization across different reasoning tasks, we avoid introducing external supervision (e.g., tools or trained value models). Our approach draws inspiration from AlphaGo
+(Silver et al.,
+2017
+)
+, where we score each intermediate node based on its contribution to the final correct answer. Consequently, actions that frequently lead to correct answers receive higher rewards, making them more likely to be selected in future MCTS tree expansions.
+We define
+Q
+​
+(
+s
+,
+a
+)
+𝑄
+𝑠
+𝑎
+Q(s,a)
+as the reward value for node
+s
+𝑠
+s
+generated under action
+a
+𝑎
+a
+.
+Initially, all unexplored nodes are assigned
+Q
+​
+(
+s
+i
+,
+a
+i
+)
+=
+0
+𝑄
+subscript
+𝑠
+𝑖
+subscript
+𝑎
+𝑖
+0
+Q(s_{i},a_{i})=0
+, leading to random tree expansions. Upon reaching the first terminal node
+n
+d
+subscript
+𝑛
+𝑑
+n_{d}
+, we compute a reward score
+Q
+​
+(
+s
+d
+,
+a
+d
+)
+𝑄
+subscript
+𝑠
+𝑑
+subscript
+𝑎
+𝑑
+Q(s_{d},a_{d})
+based on whether it reaches the correct answer.
+This score is then back-propagated to each intermediate node along the trajectory
+𝐭
+=
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+d
+𝐭
+direct-sum
+𝑥
+subscript
+𝑠
+1
+subscript
+𝑠
+2
+…
+subscript
+𝑠
+𝑑
+\mathbf{t}=x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{d}
+. Specifically, for each
+s
+i
+subscript
+𝑠
+𝑖
+s_{i}
+(for
+i
+=
+1
+,
+2
+,
+…
+,
+d
+−
+1
+𝑖
+1
+2
+…
+𝑑
+1
+i=1,2,...,d-1
+), its
+Q
+𝑄
+Q
+value is updated as follows:
+Q
+​
+(
+s
+i
+,
+a
+i
+)
+=
+Q
+​
+(
+s
+i
+,
+a
+i
+)
++
+Q
+​
+(
+s
+d
+,
+a
+d
+)
+𝑄
+subscript
+𝑠
+𝑖
+subscript
+𝑎
+𝑖
+𝑄
+subscript
+𝑠
+𝑖
+subscript
+𝑎
+𝑖
+𝑄
+subscript
+𝑠
+𝑑
+subscript
+𝑎
+𝑑
+Q(s_{i},a_{i})=Q(s_{i},a_{i})+Q(s_{d},a_{d})
+. To compute the
+Q
+​
+(
+s
+d
+,
+a
+d
+)
+𝑄
+subscript
+𝑠
+𝑑
+subscript
+𝑎
+𝑑
+Q(s_{d},a_{d})
+for the terminal node, we use the likelihood (confidence) of self-consistency majority voting as the reward value.
+Figure 4:
+The prompt example for mutual reasoning consistency.
+Solution Generation with MCTS Rollout
+. We now describe how our MCTS generates candidate reasoning trajectories. Starting from the initial root node
+s
+0
+subscript
+𝑠
+0
+s_{0}
+, we perform multiple searches consisting of
+selection
+,
+expansion
+,
+simulations
+and
+back-propagation
+. Specifically, the simulation is performed using the default
+rollout
+policy, and to achieve more accurate reward estimation, we perform multiple rollouts. To balance the exploration and exploitation, we use the well-known Upper Confidence Bounds applied to Trees (UCT)
+(Kocsis & Szepesvári,
+2006
+)
+to select each node. This selection process is mathematically represented as:
+UCT
+​
+(
+s
+,
+a
+)
+=
+Q
+​
+(
+s
+,
+a
+)
+N
+​
+(
+s
+,
+a
+)
++
+c
+​
+ln
+⁡
+N
+p
+​
+a
+​
+r
+​
+e
+​
+n
+​
+t
+​
+(
+s
+)
+N
+​
+(
+s
+,
+a
+)
+.
+UCT
+𝑠
+𝑎
+𝑄
+𝑠
+𝑎
+𝑁
+𝑠
+𝑎
+𝑐
+subscript
+𝑁
+𝑝
+𝑎
+𝑟
+𝑒
+𝑛
+𝑡
+𝑠
+𝑁
+𝑠
+𝑎
+\text{UCT}(s,a)=\frac{Q(s,a)}{N(s,a)}+c\sqrt{\frac{\ln N_{parent}(s)}{N(s,a)}}.
+where
+N
+​
+(
+s
+,
+a
+)
+𝑁
+𝑠
+𝑎
+N(s,a)
+is the number of times node
+s
+𝑠
+s
+has been visited in previous iterations, and
+N
+p
+​
+a
+​
+r
+​
+e
+​
+n
+​
+t
+​
+(
+s
+)
+subscript
+𝑁
+𝑝
+𝑎
+𝑟
+𝑒
+𝑛
+𝑡
+𝑠
+N_{parent}(s)
+represents the visiting count of the parent node of
+s
+𝑠
+s
+.
+Q
+​
+(
+s
+,
+a
+)
+𝑄
+𝑠
+𝑎
+Q(s,a)
+is the estimated reward value and will be updated through back-propagation.
+c
+𝑐
+c
+is a constant that balances exploitation and exploration.
+Once the search reaches a terminal node, either a terminal state or a predetermined maximum tree depth
+d
+𝑑
+d
+, we obtain a trajectory from the root to terminal node. We collect all trajectories from the rollout iterations as candidate solutions. The next section explains how we verify each of them.
+3.3
+Reasoning Trajectory Selection with Mutual Consistency
+In traditional MCTS, typically only one trajectory is selected as the final solution based on a specific metric, such as choosing the path with the highest reward from the rollout iterations. Unfortunately, after trying various existing methods, we found it challenging to define a single metric that reliably selects the trajectory containing the correct answer.
+Therefore, we collect all trajectories and propose mutual reasoning consistency for answer selection.
+Mutual Reasoning Consistency by Discriminator SLM
+2
+. As shown in Fig.
+2
+, in addition to the target SLM
+M
+𝑀
+M
+, we introduce another SLM
+M
+^
+^
+𝑀
+\hat{M}
+to serve as a discriminator, providing external unsupervised feedback for each candidate trajectory.
+Specifically, for
+𝐭
+=
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+d
+𝐭
+direct-sum
+𝑥
+subscript
+𝑠
+1
+subscript
+𝑠
+2
+…
+subscript
+𝑠
+𝑑
+\mathbf{t}=x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{d}
+, we mask the reasoning steps starting from a randomly sampled step
+i
+𝑖
+i
+(
+i
+<
+d
+𝑖
+𝑑
+i<d
+). We then provide the earlier reasoning trajectory
+𝐭
+=
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+i
+−
+1
+𝐭
+direct-sum
+𝑥
+subscript
+𝑠
+1
+subscript
+𝑠
+2
+…
+subscript
+𝑠
+𝑖
+1
+\mathbf{t}=x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{i-1}
+as a prompt to
+M
+^
+^
+𝑀
+\hat{M}
+to complete the remaining steps for the question. Due to the provision of the earlier
+i
+−
+1
+𝑖
+1
+i-1
+reasoning steps as a hint, we reduce the difficulty, thereby increasing the likelihood that SLM
+M
+^
+^
+𝑀
+\hat{M}
+can provide the correct answer.
+As shown in Fig.
+4
+, we compare whether the answer completed by
+M
+^
+^
+𝑀
+\hat{M}
+matches the original trajectory
+𝐭
+𝐭
+\mathbf{t}
+. If they are consistent, we consider
+t
+𝑡
+t
+as an validate trajectory for final selection.
+We provide an intuitive explanation to illustrate the rational behind our approach. Consider students solving a problem without a teacher’s feedback. A student (SLM
+1
+) unsure of their solution might ask a peer (SLM
+2
+) to review their reasoning. If the peer, given the same initial steps, arrives at the same answer, the student gains confidence in their solution. This peer verification process reflects the mutual reasoning consistency we aim to achieve.
+Final Trajectory Selection by SLM
+1
+. After applying mutual reasoning consistency to all candidate trajectories, we return to the target SLM
+M
+𝑀
+M
+to select the final trajectory from the validated ones. We compute each trajectory’s final score by multiplying its reward with the terminal node’s confidence score achieved from rollouts. The trajectory with the highest final score is chosen as the solution.
+Table 2:
+rStar greatly improves reasoning accuracy across various SLMs and tasks. rStar (generator@maj): uses majority voting for answer verification to show the MCTS generator’s effectiveness.
+Method
+LLaMA2-7B
+Mistral-7B
+LLaMA3-8B
+LLaMA3-8B-Instruct
+Phi3-mini-4k
+GSM8K
+Zero-shot CoT
+1.44
+17.89
+22.66
+68.38
+20.17
+Few-shot CoT
+12.51
+36.46
+47.23
+74.53
+83.45
+SC@maj8
+15.31
+42.91
+54.21
+78.39
+86.35
+SC@maj64
+20.77
+52.84
+64.37
+83.24
+88.02
+SC@maj128
+23.05
+57.25
+67.55
+84.69
+88.68
+ToT
+12.96
+38.89
+36.01
+69.07
+79.68
+RAP
+24.34
+56.25
+57.99
+80.59
+81.88
+rStar (generator @maj)
+27.22
+64.59
+74.38
+88.70
+90.44
+rStar
+63.91
+81.88
+85.52
+91.13
+90.67
+GSM-Hard
+Zero-shot CoT
+0.83
+5.16
+6.44
+14.94
+33.73
+Few-shot CoT
+3.71
+13.57
+13.80
+25.63
+40.63
+SC@maj8
+4.39
+17.36
+18.20
+28.51
+42.00
+SC@maj64
+6.52
+22.59
+23.73
+30.33
+44.80
+SC@maj128
+6.89
+25.01
+25.47
+31.16
+45.56
+ToT
+2.35
+11.47
+10.61
+19.64
+32.68
+RAP
+7.28
+22.52
+18.95
+29.64
+40.94
+rStar (generator @maj)
+8.64
+29.26
+26.76
+33.35
+46.55
+rStar
+18.57
+37.91
+32.97
+37.53
+46.55
+SVAMP
+Zero-shot CoT
+8.90
+26.10
+40.20
+70.90
+84.70
+Few-shot CoT
+48.10
+72.80
+76.90
+89.20
+92.80
+SC@maj8
+49.90
+74.60
+79.10
+89.20
+93.50
+SC@maj64
+54.10
+76.70
+80.70
+90.50
+93.30
+SC@maj128
+54.50
+76.60
+80.80
+90.60
+93.70
+ToT
+33.40
+56.30
+62.20
+79.80
+84.90
+RAP
+41.00
+71.80
+73.10
+85.70
+91.50
+rStar (generator @maj)
+60.30
+83.10
+86.20
+91.89
+93.80
+rStar
+74.90
+86.40
+90.00
+94.29
+94.10
+StrategyQA
+Zero-shot CoT
+52.67
+57.20
+41.48
+57.21
+54.68
+Few-shot CoT
+58.82
+65.65
+64.05
+68.41
+63.61
+SC@maj8
+59.10
+65.50
+63.76
+68.26
+64.34
+SC@maj64
+58.51
+63.61
+63.46
+67.39
+62.74
+SC@maj128
+58.37
+62.01
+63.31
+66.67
+59.53
+ToT
+45.27
+55.75
+57.64
+60.41
+40.47
+RAP
+59.68
+64.48
+63.32
+68.71
+60.26
+rStar (generator @maj)
+61.57
+69.43
+65.50
+71.47
+65.50
+rStar
+67.25
+70.31
+67.69
+71.57
+67.25
+4
+Experiments
+4.1
+Setup
+Models and Datasets
+. rStar is a general approach applicable to various LLMs and reasoning tasks. We evaluate 5 SLMs: Phi3-mini (3.8B)
+(Abdin et al.,
+2024
+)
+, LLaMA2-7B, Mistral-7B
+(Jiang et al.,
+2023
+)
+, LLaMA3-8B, and LLaMA3-8B-Instruct
+(Meta,
+2024
+)
+. We test across 5 reasoning tasks, including 4 mathematical tasks (GSM8K
+(Cobbe et al.,
+2021
+)
+, GSM-Hard
+(Gao et al.,
+2022
+)
+, MATH
+(Hendrycks et al.,
+2021
+)
+, SVAMP
+(Patel et al.,
+2021
+)
+) and one commonsense reasoning task (StrategyQA
+(Geva et al.,
+2021
+)
+).
+Implementation Details.
+In the trajectory self-generation stage, we augment each target SLM with our MCTS, performing 32 rollouts. Except for MATH, where we set the depth
+d
+𝑑
+d
+to 8, all other tasks have a
+d
+𝑑
+d
+=5. Actions
+A
+1
+subscript
+𝐴
+1
+A_{1}
+and
+A
+3
+subscript
+𝐴
+3
+A_{3}
+have a maximum of 5 nodes per depth, while the other actions have a default node count of 1. In the trajectory discrimination stage, we use Phi3-mini-4k as the discriminator, which has only 3.8B parameters, for effective inference. Moreover, the discriminator performs inference in a parallelized manner, making the verification process highly efficient.
+Notably, when Phi3 is the target SLM, it performs self-discrimination. For a given trajectory, we randomly split it between 20% and 80% of its steps, providing the first half of the steps as input to the discriminator SLM, which then completes the remaining steps.
+Detailed prompts are available in appendix
+A.3
+.
+4.2
+Main Results
+Baselines
+. We compare rStar against three strong baseline types:
+(i)
+single-round CoT prompting
+, including zero-shot CoT
+(Kojima et al.,
+2022
+)
+and few-shot CoT
+(Wei et al.,
+2022
+)
+;
+(ii)
+multi-round CoT prompting
+using the widely adopted self-consistency (SC) method
+(Wang et al.,
+2023
+)
+. We sample answers 8, 64, and 128 times, employing majority voting for answer selection; and
+(iii)
+multi-round self-improving approaches
+. For this, we select ToT
+(Yao et al.,
+2024
+)
+and RAP
+(Hao et al.,
+2023
+)
+as baselines, using BFS and MCTS for tree search, respectively. Note that the action in ToT corresponds to our action
+A
+1
+subscript
+𝐴
+1
+A_{1}
+, while RAP corresponds to our action
+A
+3
+subscript
+𝐴
+3
+A_{3}
+. For the answer selection, we follow their original implementations.
+Results on diverse reasoning benchmarks
+. We start by evaluating the effectiveness of rStar on general reasoning benchmarks. Table
+2
+compares its accuracy with state-of-the-art baselines on diverse SLMs and reasoning datasets. To demonstrate the effectiveness of our generator, we also provide the accuracy of rStar (gen. @maj), which do not apply our discriminator and use majority voting for answer verification. We highlight three key observations:
+(1)
+SLMs empowered with rStar demonstrate highly capable problem-solving abilities. For example, LLaMA2-7B originally had an accuracy of only 12.51% on GSM8K using few-shot CoT. However, with improvements from rStar, its accuracy increased to 63.91%, nearly matching the accuracy achieved with fine-tuning as shown in Fig.
+1
+. Similarly, Mistral with rStar can even outperform fine-tuned MetaMath by +4.18%. This improvement shows that SLMs already have strong reasoning capabilities but need guidance to generate and select the correct solutions.
+(2)
+rStar consistently improves the reasoning accuracy of various evaluated SLMs across different tasks to a state-of-the-art level. In contrast, none of the baseline approaches consistently perform well across all four benchmarks. For example, while SC excels in three mathematical tasks, it is less effective on the logical reasoning task of StrategyQA. Specifically, SC with more sampling can even lower the score on StrategyQA. RAP performs better than SC on StrategyQA but falls short compared to SC on most mathematical reasoning tasks.
+(3)
+Even without our proposed discriminator for reasoning trajectory verification, our MCTS generator demonstrates greater effectiveness in improving reasoning accuracy for SLMs compared to existing multi-round inference baselines. For example, rStar (generator @maj) achieves up to 2.88%-16.39% higher accuracy than RAP, 10.60%- 38.37% higher accuracy than ToT, and 1.69% - 7.34% higher accuracy than SC on the GSM8K dataset.
+Table 3:
+Reasoning performance comparison on the challenging MATH-500 dataset. Due to the extensive LaTeX syntax in the dataset, which is challenging for pre-trained LLMs in instruction following, we evaluate only on LLaMA3-8B-instruct and Phi3-Mini-4k-Instruct.
+Method
+LLaMA3-8b-Instruct
+Phi3-mini-4k
+Zeroshot CoT
+5.80
+3.60
+Fewshot CoT
+17.80
+32.20
+SC@maj8
+30.00
+40.40
+SC@maj64
+33.00
+45.20
+SC@maj128
+33.80
+45.60
+ToT
+13.60
+18.20
+RAP
+18.80
+27.80
+rStar (generator @maj)
+38.30
+48.40
+rStar
+42.94
+48.60
+Figure 5:
+Performance comparison on the GSM8K dataset under different number of rollouts. rStar can significantly improve reasoning accuracy with just 2 rollouts.
+Results on challenging mathematical dataset
+. We also evaluate the effectiveness of rStar on more challenging mathematical datasets. In particular, we select the GSM-Hard and MATH datasets. Following
+(Wang et al.,
+2024b
+; Lightman et al.,
+2023
+)
+, we use MATH-500, a subset of representative problems from the MATH dataset, to speedup the evaluation. As shown in Table
+2
+and Table
+3
+, rStar
+is capable of significantly improve the reasoning accuracy of SLMs on these challenging mathematical datasets.
+Remarkably, when compared to SOTA baselines, we observe a significant improvements of up to +12.9% and +9.14% on GSM-Hard and MATH-500, respectively.
+4.3
+Ablation Study
+Table 4:
+Ablation study on the effectiveness of our MCTS generator. Ours+self-eval: we apply self-evaluation to prompt model for self-rewarding each intermediate action in our generator.
+Generator
+LLaMA3-8B
+LLaMA3-8B-Instruct
+GSM8K
+StrategyQA
+GSM8K
+StrategyQA
+Answer verification
+Answer verification
+Answer verification
+Answer verification
+Maj
+Ours
+Maj
+Ours
+Maj
+Ours
+Maj
+Ours
+RAP
+56.56
+57.31
+62.30
+64.63
+81.35
+84.69
+69.43
+70.60
+SC (@128)
+67.55
+85.06
+63.31
+65.65
+84.69
+89.99
+66.67
+68.56
+Ours+Self-eval
+70.28
+82.18
+65.07
+66.22
+88.07
+89.92
+69.28
+69.43
+Ours
+74.38
+85.52
+65.50
+67.69
+88.70
+91.13
+71.47
+71.57
+Table 5:
+Ablation study on discriminator effectiveness. We evaluate accuracy on GSM8K.
+Left
+: Our discriminator consistently outperforms others in verifying solution trajectories generated by different generators.
+Right
+: The ablation study on the choice of discriminator model.
+Discriminator
+LLaMA3-8B
+LLaMA3-8B-Instruct
+Generator
+Generator
+SC
+Ours
+SC
+Ours
+Maj
+67.55
+74.38
+84.69
+88.70
+Self-verification
+74.00
+75.52
+83.02
+86.63
+Ours
+85.06
+85.52
+89.99
+91.13
+Model
+Discriminator SLM
+Accuracy
+LLaMA3-8B-Instruct
+Maj
+88.70
+LLaMA3-8B-Instruct
+88.78
+LLaMA3.1-8B-Instruct
+89.52
+Phi3-Mini-Instruct
+91.13
+GPT-4 (2024-05-01)
+92.57
+Effectiveness under different rollouts
+. rStar uses a rollout policy for MCTS tree expansion. More rollouts generate more candidate solution trajectories but increase inference cost. In Fig.
+5
+, we compare the accuracy of SC, RAP, and our rStar across different rollouts on GSM8K. For SC, we sample solutions based on each number of rollouts and use majority voting to select the answer. We highlight two key observations:
+(1)
+Even with just 2 rollouts, rStar significantly improves reasoning accuracy for SLMs, demonstrating its effectiveness;
+(2)
+Both rStar and SC benefit from more rollouts, whereas RAP tends to saturate and even decline after 4 rollouts on LLaMA3-8B-Instruct. One reason is that the single-type action space in RAP limits the effective MCTS exploration.
+The effectiveness of MCTS generator
+. We compare our MCTS generator with three baselines: (i) the MCTS generator used in RAP; (ii) SC with 128 randomly sampled solutions; and (iii) our generator with Self-evaluation, a popular technique that self-evaluates the reward score for each action. Baseline (iii) specifically evaluates the effectiveness of our reward function.
+To isolate the impact of answer verification methods, each generator is evaluated under both majority voting and our discriminator for trajectory selection. As shown in Table
+4
+, our generator consistently outperforms the baseline generators across different answer verification methods. More, we demonstrate the effectiveness of our SLM-tailored reward function, as self-evaluation reduces our generator’s accuracy.
+The effectiveness of discriminator
+. We setup two experiments for evaluation. First, we compare our discrimination approach with two baselines: the majority voting and self-verification
+(Weng et al.,
+2023
+)
+. Specifically, we follow the key idea in
+Weng et al. (
+2023
+)
+to prompt the SLM (i.e., generator SLM) to self-verify the correctness of each trajectory. To demonstrate the generalization ability of our discriminator, we used candidate solutions from different generators for evaluation.
+As shown in Table
+5
+(Left), our discriminator significantly improves reasoning accuracy when performing answer verification on trajectories generated by different generators. Similar to the previous self-evaluation experiment, self-verification on SLMs is ineffective.
+Second, we study the impact of discriminator model selection. Our current discriminator models are all Phi3-Mini-Instruct. We tested various LLMs, both stronger and weaker, as discriminators for LLaMA3-8B-Instruct. As shown in Table
+5
+(Right), the choice of discriminator model generally does not affect the effectiveness of our mutual reasoning consistency for answer verification. Notably, using the powerful GPT-4 as the discriminator only slightly improves performance (91.13% to 92.57%), demonstrating that mutual reasoning consistency can effectively verify answers using SLMs.
+5
+Conclusion
+In this work, we present rStar, a generator-discriminator self-play approach that significantly grow the reasoning capabilities for SLMs at the inference time. Our approach reveals that SLMs, such as LLaMA2-7B, already exhibit strong reasoning capabilities prior to domain specialized supervised fine-tuning. rStar achieves state-of-the-art performance across five SLMs and five diverse reasoning tasks, substantially outperforming existing multi-round prompting and self-improvement approaches. Furthermore, we conduct extensive ablation studies and analysis, contributing to the development of more advanced SLM self-improved reasoning.
+References
+Abdin et al. (2024)
+Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah,
+Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl,
+et al.
+Phi-3 technical report: A highly capable language model locally on
+your phone.
+arXiv preprint arXiv:2404.14219
+, 2024.
+Besta et al. (2024)
+Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal
+Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert
+Niewiadomski, Piotr Nyczyk, et al.
+Graph of thoughts: Solving elaborate problems with large language
+models.
+In
+Proceedings of the AAAI Conference on Artificial
+Intelligence
+, volume 38, pp.  17682–17690, 2024.
+Brown et al. (2024)
+Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le,
+Christopher Ré, and Azalia Mirhoseini.
+Large language monkeys: Scaling inference compute with repeated
+sampling.
+arXiv preprint arXiv:2407.21787
+, 2024.
+Chen et al. (2024a)
+Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan.
+Alphamath almost zero: process supervision without process,
+2024a.
+Chen et al. (2022)
+Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen.
+Program of thoughts prompting: Disentangling computation from
+reasoning for numerical reasoning tasks.
+arXiv preprint arXiv:2211.12588
+, 2022.
+Chen et al. (2024b)
+Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.
+Self-play fine-tuning converts weak language models to strong
+language models.
+arXiv preprint arXiv:2401.01335
+, 2024b.
+Cobbe et al. (2021)
+Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz
+Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano,
+et al.
+Training verifiers to solve math word problems.
+arXiv preprint arXiv:2110.14168
+, 2021.
+Ding et al. (2023)
+Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin,
+Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang.
+Everything of thoughts: Defying the law of penrose triangle for
+thought generation.
+arXiv preprint arXiv:2311.04254
+, 2023.
+Feng et al. (2023)
+Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang.
+Alphazero-like tree-search can guide large language model decoding
+and training.
+arXiv preprint arXiv:2309.17179
+, 2023.
+Forsman (2024)
+Anton Forsman.
+Analyzing the performance of self-refine on different large language
+models.
+2024.
+URL
+https://github.com/anforsm/self-refine/blob/main/report.pdf
+.
+Gao et al. (2022)
+Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie
+Callan, and Graham Neubig.
+Pal: Program-aided language models.
+arXiv preprint arXiv:2211.10435
+, 2022.
+Gero et al. (2023)
+Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley,
+Jianfeng Gao, and Hoifung Poon.
+Self-verification improves few-shot clinical information extraction.
+arXiv preprint arXiv:2306.00024
+, 2023.
+Geva et al. (2021)
+Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan
+Berant.
+Did aristotle use a laptop? a question answering benchmark with
+implicit reasoning strategies.
+Transactions of the Association for Computational Linguistics
+,
+9:346–361, 2021.
+URL
+https://huggingface.co/datasets/ChilleD/StrategyQA
+.
+Gou et al. (2023)
+Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan,
+Weizhu Chen, et al.
+Tora: A tool-integrated reasoning agent for mathematical problem
+solving.
+arXiv preprint arXiv:2309.17452
+, 2023.
+Hao et al. (2023)
+Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and
+Zhiting Hu.
+Reasoning with language model is planning with world model.
+arXiv preprint arXiv:2305.14992
+, 2023.
+Hendrycks et al. (2021)
+Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric
+Tang, Dawn Song, and Jacob Steinhardt.
+Measuring mathematical problem solving with the math dataset.
+arXiv preprint arXiv:2103.03874
+, 2021.
+Huang et al. (2023)
+Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu,
+Xinying Song, and Denny Zhou.
+Large language models cannot self-correct reasoning yet.
+arXiv preprint arXiv:2310.01798
+, 2023.
+Jiang et al. (2023)
+Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford,
+Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel,
+Guillaume Lample, Lucile Saulnier, et al.
+Mistral 7b.
+arXiv preprint arXiv:2310.06825
+, 2023.
+Kang et al. (2024)
+Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, and Boxing Chen.
+Mindstar: Enhancing math reasoning in pre-trained llms at inference
+time.
+arXiv preprint arXiv:2405.16265
+, 2024.
+Khot et al. (2022)
+Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter
+Clark, and Ashish Sabharwal.
+Decomposed prompting: A modular approach for solving complex tasks.
+arXiv preprint arXiv:2210.02406
+, 2022.
+Kocsis & Szepesvári (2006)
+Levente Kocsis and Csaba Szepesvári.
+Bandit based monte-carlo planning.
+volume 2006, pp.  282–293, 09 2006.
+ISBN 978-3-540-45375-8.
+doi:
+10.1007/11871842_29
+.
+Kojima et al. (2022)
+Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke
+Iwasawa.
+Large language models are zero-shot reasoners.
+Advances in neural information processing systems
+,
+35:22199–22213, 2022.
+Li et al. (2024)
+Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng
+Zhang, and Houwen Peng.
+Common 7b language models already possess strong math capabilities.
+arXiv preprint arXiv:2403.04706
+, 2024.
+Lightman et al. (2023)
+Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy
+Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
+Let’s verify step by step.
+arXiv preprint arXiv:2305.20050
+, 2023.
+Madaan et al. (2024)
+Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah
+Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.
+Self-refine: Iterative refinement with self-feedback.
+Advances in Neural Information Processing Systems
+, 36, 2024.
+Meta (2024)
+Meta.
+Introducing meta llama3: The most capable openly available llm to
+date, 2024.
+URL
+https://ai.meta.com/blog/meta-llama-3/
+.
+Patel et al. (2021)
+Arkil Patel, Satwik Bhattamishra, and Navin Goyal.
+Are nlp models really able to solve simple math word problems?
+In
+Proceedings of the 2021 Conference of the North American
+Chapter of the Association for Computational Linguistics: Human Language
+Technologies
+, pp.  2080–2094, 2021.
+Roy & Roth (2015)
+Subhro Roy and Dan Roth.
+Solving General Arithmetic Word Problems.
+In
+Proc. of the Conference on Empirical Methods in Natural
+Language Processing (EMNLP)
+, 2015.
+URL
+http://cogcomp.org/papers/arithmetic.pdf
+.
+Silver et al. (2017)
+David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew
+Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore
+Graepel, et al.
+Mastering chess and shogi by self-play with a general reinforcement
+learning algorithm.
+arXiv preprint arXiv:1712.01815
+, 2017.
+Snell et al. (2024)
+Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
+Scaling llm test-time compute optimally can be more effective than
+scaling model parameters, 2024.
+URL
+https://arxiv.org/abs/2408.03314
+.
+Valmeekam et al. (2022)
+Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati.
+Large language models still can’t plan (a benchmark for LLMs on
+planning and reasoning about change).
+In
+NeurIPS 2022 Foundation Models for Decision Making
+Workshop
+, 2022.
+URL
+https://openreview.net/forum?id=wUU-7XTL5XO
+.
+Wang et al. (2024a)
+Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui
+Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li.
+Mathcoder: Seamless code integration in LLMs for enhanced
+mathematical reasoning.
+In
+The Twelfth International Conference on Learning
+Representations
+, 2024a.
+URL
+https://openreview.net/forum?id=z8TW0ttBPp
+.
+Wang et al. (2024b)
+Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen,
+Y. Wu, and Zhifang Sui.
+Math-shepherd: Verify and reinforce llms step-by-step without human
+annotations, 2024b.
+Wang et al. (2023)
+Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang,
+Aakanksha Chowdhery, and Denny Zhou.
+Self-consistency improves chain of thought reasoning in language
+models.
+In
+The Eleventh International Conference on Learning
+Representations
+, 2023.
+URL
+https://openreview.net/forum?id=1PL1NIMMrw
+.
+Wei et al. (2022)
+Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
+Le, Denny Zhou, et al.
+Chain-of-thought prompting elicits reasoning in large language
+models.
+Advances in Neural Information Processing Systems
+,
+35:24824–24837, 2022.
+Weng et al. (2023)
+Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun,
+Kang Liu, and Jun Zhao.
+Large language models are better reasoners with self-verification.
+2023.
+Wu et al. (2024)
+Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang.
+Large language models can self-correct with minimal effort.
+arXiv preprint arXiv:2405.14092
+, 2024.
+Yao et al. (2024)
+Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and
+Karthik Narasimhan.
+Tree of thoughts: Deliberate problem solving with large language
+models.
+Advances in Neural Information Processing Systems
+, 36, 2024.
+Zhang et al. (2024)
+Di Zhang, Jiatong Li, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli
+Ouyang.
+Accessing gpt-4 level mathematical olympiad solutions via monte carlo
+tree self-refine with llama-3 8b.
+arXiv preprint arXiv:2406.07394
+, 2024.
+Zheng et al. (2023)
+Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H Chi,
+Quoc V Le, and Denny Zhou.
+Take a step back: Evoking reasoning via abstraction in large language
+models.
+arXiv preprint arXiv:2310.06117
+, 2023.
+Zhou et al. (2023)
+Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu,
+Anya Jia, Linqi Song, Mingjie Zhan, et al.
+Solving challenging math word problems using gpt-4 code interpreter
+with code-based self-verification.
+arXiv preprint arXiv:2308.07921
+, 2023.
+Zhou et al. (2022)
+Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi
+Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, et al.
+Least-to-most prompting enables complex reasoning in large language
+models.
+In
+The Eleventh International Conference on Learning
+Representations
+, 2022.
+Zhou et al. (2024)
+Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V Le, Ed H
+Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng.
+Self-discover: Large language models self-compose reasoning
+structures.
+arXiv preprint arXiv:2402.03620
+, 2024.
+Appendix A
+Appendix
+A.1
+Experiments to evaluate the self-rewarding in SLMs
+Table 6:
+Analysis on the effectiveness of SLMs’ self-rewarding. The original
+r
+1
+subscript
+𝑟
+1
+r_{1}
+is a self-evaluation of the helpfulness of the new proposed subquestion, while
+r
+2
+subscript
+𝑟
+2
+r_{2}
+measures the confidence in answering the subquestion through self-consistency majority voting. Results show that replacing the self-evaluated
+r
+1
+subscript
+𝑟
+1
+r_{1}
+to random values does not significantly impact the final reasoning performance.
+Method
+LLaMA2-7B
+Mistral
+GSM8K
+RAP
+24.34
+56.25
+RAP + random
+r
+1
+subscript
+𝑟
+1
+r_{1}
+22.90
+55.50
+RAP + random
+r
+2
+subscript
+𝑟
+2
+r_{2}
+22.67
+49.66
+Multiarith
+RAP
+57.22
+91.11
+RAP + random
+r
+1
+subscript
+𝑟
+1
+r_{1}
+52.78
+90.56
+RAP + random
+r
+2
+subscript
+𝑟
+2
+r_{2}
+47.22
+81.11
+Ablation study on self-rewarding in RAP
+. RAP rewards both intermediate and terminal nodes. For each node generated by its action, it combines two scores,
+r
+1
+subscript
+𝑟
+1
+r_{1}
+and
+r
+2
+subscript
+𝑟
+2
+r_{2}
+, to determine the final reward score. Formally,
+r
+=
+r
+1
+×
+r
+2
+𝑟
+subscript
+𝑟
+1
+subscript
+𝑟
+2
+r=r_{1}\times r_{2}
+.
+r
+1
+subscript
+𝑟
+1
+r_{1}
+is a self-evaluation score that evaluates the LLM’s own estimation of the helpfulness of the current node. Specifically, it prompts the LLM with the question "
+Is the new question useful
+?".
+r
+2
+subscript
+𝑟
+2
+r_{2}
+is the confidence of correctly answering the proposed new question, measured by self-consistency majority voting.
+To evaluate the effectiveness of self-rewarding in RAP, we replace
+r
+1
+subscript
+𝑟
+1
+r_{1}
+and
+r
+2
+subscript
+𝑟
+2
+r_{2}
+with random values sampled from (0,1)and re-run RAP on LLaMA2-7B and Mistral-7B. We select a challenging dataset, GSM8K and an easy mathematical reasoning dataset, Multiarith
+(Roy & Roth,
+2015
+)
+, for evaluation.
+Table
+6
+compares the results with original RAP. We can see that replacing
+r
+1
+subscript
+𝑟
+1
+r_{1}
+with random values has minimal impact on RAP’s performance across different SLMs and datasets. However, replacing
+r
+2
+subscript
+𝑟
+2
+r_{2}
+with random values result in a noticeable drop in accuracy on Mistral and Multiarith. This indicates that self-evaluation
+r
+1
+subscript
+𝑟
+1
+r_{1}
+has minimal effect, suggesting that LLaMA2-7B and Mistral are essentially performing near-random self-evaluations.
+A.2
+Discussions
+Discussions on the importance of generator and discriminator
+. In our experiments, we found that on certain SLMs, the discriminator yields more significant improvement than the generator. For instance, on LLaMA2-7B, rStar (generator @maj) can improves accuracy by +4.17% on GSM8K, while our discriminator can further boosts accuracy by +36.69%. However, both the generator and discriminator are crucial to final performance. The generator must effectively produce the correct solution first, and the discriminator then provide reasonable feedback to select the correct solution.
+The importance of the generator and discriminator varies based on the SLM’s solution generation effectiveness. For LLaMA2-7B, the proportion of correct solution trajectories is low, leading to a lower majority voting score. In this case, the discriminator is crucial for identifying the correct solution, significantly boosting accuracy. Conversely, for stronger models like LLaMA3-8B-instruct, our generator produces a higher proportion of correct solutions. Therefore, improving the generator results in greater accuracy improvements.
+Table 7:
+Inference costs of rStar on GSM8K. We show the average number of inferences and generated tokens required to solve a question.
+LLaMA2-7B
+Mistral
+Avg. calls
+166.81
+148.90
+Avg. generated tokens
+367.1k
+348.6k
+Inference cost
+. rStar grows SLMs reasoning capabilities at inference time. The primary inference cost arises from our MCTS self-generator. Table
+7
+shows the average number of inferences and tokens generated for solving a GSM8K question after 32 rollouts.
+On LLaMA2-7B and Mistral, this averages 166 and 148 model calls to solve a question, respectively. Currently, completing the 32 rollouts for the entire GSM8K test set takes about 4.5 days on a single A100 GPU per model. These costs can be significantly reduced by distributing tasks across multiple GPUs or batching model calls within each rollout.
+A.3
+Prompt examples in MTCS
+Action 1. Propose an one-step thought.
+### Instruction:
+There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
+### Response:
+Let’s think step by step.
+Step 1: Identify the initial number of trees. The problem states there are 15 trees in the grove.
+Step 2: Identify the final number of trees. The problem states there will be 21 trees after the workers are done planting.
+Step 3: Subtract the initial number of trees from the final number of trees to find out how many trees were planted.
+Step 4: Therefore, the grove workers planted 21 (final number of trees) - 15 (initial number of trees) = 6 trees today.
+Step 5: The answer is 6.
+### Instruction:
+If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
+### Response:
+Let’s think step by step.
+Step 1: Start with the number of cars that are already in the parking lot, which is 3 cars.
+Step 2: Add the number of cars that arrive, which is 2 cars.
+Step 3: Add the numbers together. there are 3 cars + 2 cars = 5 cars in the parking lot.
+Step 4: The answer is 5.
+### Instruction:
+Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
+### Response:
+Let’s think step by step.
+Step 1: Add the number of chocolates Leah and her sister had initially. Leah had 32 chocolates and her sister had 42 chocolates. So, they had 32 + 42 = 74 chocolates in total.
+Step 2: Subtract the number of chocolates they ate from the total number of chocolates they had. They ate 35 chocolates. So, they have 74 - 35 = 39 chocolates left.
+Step 3: The answer is 39.
+### Instruction:
+Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
+### Response:
+Let’s think step by step.
+Step 1: Identify the initial number of lollipops. The problem states Jason had 20 lollipops.
+Step 2: Identify the final number of lollipops. The problem states Jason now has 12 lollipops.
+Step 3: Subtract the final number of lollipops from the initial number to find out how many lollipops Jason gave to Denny.
+Step 4: Therefore, Jason gave Denny 20 (initial number of lollipops) - 12 (final number of lollipops) = 8 lollipops.
+Step 5: The answer is 8.
+###
+Instruction:
+{user question}
+###
+Response:
+Let’s think step by step.
+Action 2: Propose the remaining thought steps /A4: Answer the sub-question again.
+### Instruction:
+There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
+### Response:
+Let’s think step by step. There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6. The answer is: 6.
+### Instruction:
+If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
+### Response:
+Let’s think step by step. There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5. The answer is: 5.
+### Instruction:
+Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
+### Response:
+Let’s think step by step. Originally, Leah had 32 chocolates. Her sister had 42. So in total they had 32 + 42 = 74. After eating 35, they had 74 - 35 = 39. The answer is: 39.
+### Instruction:
+Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
+### Response:
+Let’s think step by step. Jason started with 20 lollipops. Then he had 12 after giving some to Denny. So he gave Denny 20 - 12 = 8. The answer is: 8.
+### Instruction:
+Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
+### Response:
+Let’s think step by step. Shawn started with 5 toys. If he got 2 toys each from his mom and dad, then that is 4 more toys. 5 + 4 = 9. The answer is: 9.
+### Instruction:
+There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?
+### Response:
+Let’s think step by step. There were originally 9 computers. For each of 4 days, 5 more computers were added. So 5 * 4 = 20 computers were added. 9 + 20 is 29. The answer is: 29.
+### Instruction:
+Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday?
+### Response:
+Let’s think step by step. Michael started with 58 golf balls. After losing 23 on tuesday, he had 58 - 23 = 35. After losing 2 more, he had 35 - 2 = 33 golf balls. The answer is: 33.
+### Instruction:
+Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
+### Response:
+Let’s think step by step. Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 x 3 = 15 dollars. So she has 23 - 15 dollars left. 23 - 15 is 8. The answer is: 8.
+###
+Instruction:
+{user question}
+###
+Response
+:
+Action 3: Propose next sub-question along with its answer.
+Given a question, please decompose it into sub-questions. For each sub-question, please answer it in a complete sentence, ending with "The answer is <a numeric answer>". When the original question is answerable, please start the subquestion with "Now we can answer the question: <original question>".
+Question 1: Four years ago, Kody was only half as old as Mohamed. If Mohamed is currently twice as 30 years old, how old is Kody?
+Question 1.1: How old is Mohamed currently?
+Answer 1.1: Mohamed is twice as old as 30 years, which means he is 30 * 2 = 60 years old.
+Question 1.2: What was Kody’s age four years ago, given that it was half of Mohamed’s age at that time?
+Answer 1.2: Four years ago, Mohamed was 60 - 4 = 56 years old, so Kody was half of that, which is 56 / 2 = 28 years old.
+Question 1.3: Now we can answer the question: How old is Kody?
+Answer 1.3: Kody is currently 28 + 4 = 32 years old. The answer is 32.
+Question 2: On a moonless night, three fireflies danced in the evening breeze. They were joined by four less than a dozen more fireflies before two of the fireflies flew away. How many fireflies remained?
+Question 2.1: How many fireflies joined?
+Answer 2.1: The fireflies were joined by four less than a dozen more fireflies, which are 12 - 4 = 8 fireflies. The answer is 8.
+Question 2.2: Now we can answer the question: How many fireflies remained?
+Answer 2.2: Three fireflies were dancing originally. They were joined by 8 fireflies before two of them flew away. So there were 3 + 8 - 2 = 9 remaining. The answer is 9.
+Question 3: Ali has four $10 bills and six $20 bills that he saved after working for Mr. James on his farm. Ali gives her sister half of the total money he has and uses 3/5 of the remaining amount of money to buy dinner. Calculate the amount of money he has after buying the dinner.
+Question 3.1: How much money does Ali have after giving half of his total money to his sister?
+Answer 3.1: Ali initially has four $10 bills and six $20 bills, totaling 4 * 10 + 6 * 20 = 160 dollars. Giving half of this to his sister leaves him with 160 / 2 = 80 dollars. The answer is 80.
+Question 3.2: How much money does Ali spend on dinner?
+Answer 3.2: Ali uses 3/5 of his remaining money, which is 80 dollars, to buy dinner. Therefore, he spends 80 * 3/5 = 48 dollars on dinner. The answer is 48.
+Question 3.3: Now we can answer the question: How much money does Ali have after buying the dinner?
+Answer 3.3: After buying the dinner, Ali has 80 - 48 = 32 dollars left. The answer is 32.
+Question 4: A car is driving through a tunnel with many turns. After a while, the car must travel through a ring that requires a total of 4 right-hand turns. After the 1st turn, it travels 5 meters. After the 2nd turn, it travels 8 meters. After the 3rd turn, it travels a little further and at the 4th turn, it immediately exits the tunnel. If the car has driven a total of 23 meters around the ring, how far did it have to travel after the 3rd turn?
+Question 4.1: How far did the car travel except for the 3rd turn?
+Answer 4.1: It travels 5 meters after the 1st, 8 meters after the 2nd, and 0 meters after the 4th turn. It’s a total of 5 + 8 + 0 = 13 meters. The answer is 13.
+Question 4.2: Now we can answer the question: How far did the car have to travel after the 3rd turn?
+Answer 4.2: The car has driven a total of 23 meters around the ring. It travels 13 meters except for the 3rd turn. So it has to travel 23 - 13 = 10 meters after the 3rd turn. The answer is 10.
+Question 5: {user question}
+Action 5: Rephrase the question/sub-question.
+You are an AI assistant to help me rephrase questions by splitting the question context into conditions. In your rephrased question, remember to fully express the information in the original question.
+Original Question: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
+Rephrased Question: Given a list of conditions, please answer the question. Condition 1: Olivia starts with $23. Condition 2: She buys five bagels, each costing $3. Question: How much money does Olivia have remaining after her purchase?
+Original Question: Michael had 58 golf balls. On Tuesday, he lost 23 golf balls. On Wednesday, he lost 2 more. How many golf balls did he have at the end of Wednesday?
+Rephrased Question: Given a list of conditions, please answer the question. Condition 1: Michael initially has 58 golf balls. Condition 2: On Tuesday, he loses 23 golf balls. Condition 3: On Wednesday, he loses 2 additional golf balls. Question: What is the total number of golf balls Michael has left at the end of Wednesday?
+Original Question: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?
+Rephrased Question: Given a list of conditions, please answer the question. Condition 1: Angelo and Melanie need to study 2 textbook chapters and 4 worksheets. Condition 2: They allocate 3 hours per textbook chapter and 1.5 hours per worksheet. Condition 3: Their daily study limit is 4 hours, with a 10-minute break every hour, three 10-minute snack breaks, and a 30-minute lunch break each day. Question: Over the next week, for how many days should they plan to study to cover all their materials?
+Original Question: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
+Rephrased Question: Given a list of conditions, please answer the question. Condition 1: Leah has 32 chocolates. Condition 2: Her sister has 42 chocolates. Condition 3: Together, they consume 35 chocolates. Question: How many chocolates remain between them after they have eaten some?
+Original Question: There were nine computers in the server room. Five more computers were installed each day, from Monday to Thursday. How many computers are now in the server room?
+Rephrased Question: Given a list of conditions, please answer the question. Condition 1: Initially, there are nine computers in the server room. Condition 2: Each day, from Monday to Thursday, five additional computers are installed. Question: What is the total number of computers in the server room after these installations?
+Original Question: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?
+Rephrased Question: Given a list of conditions, please answer the question. Condition 1: Jason starts with 20 lollipops. Condition 2: After giving some lollipops to Denny, Jason has 12 lollipops left. Question: How many lollipops did Jason give to Denny?
+Original Question: Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He rearranged five of these boxes into packages of six highlighters each and sold them for $3 per package. He sold the rest of the highlighters separately at the rate of three pens for $2. How much profit did he make in total, in dollars?
+Rephrased Question: Given a list of conditions, please answer the question. Condition 1: Sam purchases a dozen boxes of highlighters, with each box containing 30 pens, at $10 per box. Condition 2: He repackages five boxes into packages of six highlighters, selling each package for $3. Condition 3: He sells the remaining highlighters at a rate of three for $2. Question: What is Sam’s total profit from these transactions?
+Original Question: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
+Rephrased Question: Given a list of conditions, please answer the question. Condition 1: Initially, there are 15 trees in the grove. Condition 2: Grove workers will add more trees to the grove today. Condition 3: After planting, the total number of trees in the grove will increase to 21. Question: How many trees did the grove workers plant today?
+Original Question: {user question}
+Rephrased Question:
+◄
+Feeling
+lucky?
+Conversion
+report
+Report
+an issue
+View original
+on arXiv
+►

research/notes/240806195-mutual-reasoning-makes-smaller-llms-stronger-problem-solvers.md

@@ -0,0 +1,191 @@
+---
+title: '[2408.06195] Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers'
+id: 240806195-mutual-reasoning-makes-smaller-llms-stronger-problem-solvers
+tags:
+- deepread
+created: '2026-06-10T00:39:56.384617Z'
+source: https://arxiv.org/abs/2408.06195
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:39:56.384488Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2408.06195] Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers
+Computer Science > Computation and Language
+arXiv:2408.06195
+(cs)
+[Submitted on 12 Aug 2024]
+Title:
+Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers
+Authors:
+Zhenting Qi
+,
+Mingyuan Ma
+,
+Jiahang Xu
+,
+Li Lyna Zhang
+,
+Fan Yang
+,
+Mao Yang
+View a PDF of the paper titled Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers, by Zhenting Qi and 5 other authors
+View PDF
+HTML (experimental)
+Abstract:
+This paper introduces rStar, a self-play mutual reasoning approach that significantly improves reasoning capabilities of small language models (SLMs) without fine-tuning or superior models. rStar decouples reasoning into a self-play mutual generation-discrimination process. First, a target SLM augments the Monte Carlo Tree Search (MCTS) with a rich set of human-like reasoning actions to construct higher quality reasoning trajectories. Next, another SLM, with capabilities similar to the target SLM, acts as a discriminator to verify each trajectory generated by the target SLM. The mutually agreed reasoning trajectories are considered mutual consistent, thus are more likely to be correct. Extensive experiments across five SLMs demonstrate rStar can effectively solve diverse reasoning problems, including GSM8K, GSM-Hard, MATH, SVAMP, and StrategyQA. Remarkably, rStar boosts GSM8K accuracy from 12.51% to 63.91% for LLaMA2-7B, from 36.46% to 81.88% for Mistral-7B, from 74.53% to 91.13% for LLaMA3-8B-Instruct. Code will be available at
+this https URL
+.
+Subjects:
+Computation and Language (cs.CL)
+Cite as:
+arXiv:2408.06195
+[cs.CL]
+(or
+arXiv:2408.06195v1
+[cs.CL]
+for this version)
+https://doi.org/10.48550/arXiv.2408.06195
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Li Lyna Zhang [
+view email
+]
+[v1]
+Mon, 12 Aug 2024 14:42:13 UTC (1,140 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers, by Zhenting Qi and 5 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.CL
+< prev
+|
+next >
+new
+|
+recent
+|
+2024-08
+Change to browse by:
+cs
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/241020285-swe-search-enhancing-software-agents-with-monte-carlo-tree-search-and-2.md

@@ -0,0 +1,2144 @@
+---
+title: '[2410.20285] SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search
+  and Iterative Refinement'
+id: 241020285-swe-search-enhancing-software-agents-with-monte-carlo-tree-search-and-2
+tags:
+- deepread
+created: '2026-06-10T00:41:20.614811Z'
+source: https://ar5iv.labs.arxiv.org/html/2410.20285
+source_domain: ar5iv.labs.arxiv.org
+fetched_at: '2026-06-10T00:41:20.614654Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2410.20285] SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement
+SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement
+Antonis Antoniades
+1∗
+, Albert Örwall
+2
+,
+Kexun Zhang
+3
+,
+Yuxi Xie
+4
+, Anirudh Goyal
+5
+, William Wang
+1
+1
+University of California, Santa Barbara,
+2
+Moatless AI,
+3
+Carnegie Mellon University,
+4
+National University of Singapore,
+5
+Mila
+Denotes equal contribution.
+Correspondence to:
+antonis@ucsb.edu
+,
+albert@moatless.ai
+.
+Code:
+github.com/aorwall/moatless-tree-search
+Abstract
+Software engineers operating in complex and dynamic environments must continuously adapt to evolving requirements, learn iteratively from experience, and reconsider their approaches based on new insights. However, current large language model (LLM)-based software agents often rely on rigid processes and tend to repeat ineffective actions without the capacity to evaluate their performance or adapt their strategies over time. To address these challenges, we propose SWE-Search, a multi-agent framework that integrates Monte Carlo Tree Search (MCTS) with a self-improvement mechanism to enhance software agents’ performance on repository-level software tasks. SWE-Search extends traditional MCTS by incorporating a hybrid value function that leverages LLMs for both numerical value estimation and qualitative evaluation. This enables self-feedback loops where agents iteratively refine their strategies based on both quantitative numerical evaluations and qualitative natural language assessments of pursued trajectories. The framework includes a SWE-Agent for adaptive exploration, a Value Agent for iterative feedback, and a Discriminator Agent that facilitates multi-agent debate for collaborative decision-making. Applied to the SWE-bench benchmark, our approach demonstrates a 23% relative improvement in performance across five models compared to standard open-source agents without MCTS. Our analysis reveals how performance scales with increased search depth and identifies key factors that facilitate effective self-evaluation in software agents. This work highlights the potential of self-evaluation driven search techniques to enhance agent reasoning and planning in complex, dynamic software engineering environments.
+1
+Introduction
+Software engineering is a complex and iterative process involving exploration, problem-solving, and decision-making under uncertainty. Tasks such as debugging, feature development, and code refactoring require continuous assessment of different approaches, frequent backtracking, and the incorporation of new information. While machine learning has made progress in automating parts of this workflow
+(Li et al.,
+2022
+; OpenAI et al.,
+2024
+; Ouyang et al.,
+2022
+; Yang et al.,
+2024b
+)
+, replicating the adaptive and strategic behavior of human engineers remains a significant challenge. This is due to the inherently non-linear and iterative nature of software engineering, where engineers dynamically explore various solutions, refine strategies based on feedback, and collaborate to identify the most effective path forward. Current large language model (LLM)-based software agents
+(Xia et al.,
+2024
+; Zhang et al.,
+2024d
+)
+, while powerful, often struggle with complex, long-horizon tasks that require adaptive strategies and flexible reassessment over time. These agents can become trapped in repetitive patterns, limiting their effectiveness in tackling more intricate software engineering problems.
+To address these challenges, we introduce
+SWE-Search
+, a multi-agent system that replicates the adaptability, iterative learning, and collaborative decision-making of human engineers. SWE-Search is designed to address three critical needs in software engineering:
+Flexible Exploration and Adaptation
+: Engineering problems often require exploring multiple approaches and adapting strategies based on evolving information
+(Li et al.,
+2022
+)
+. SWE-Search’s SWE-Agent operates in a flexible state space, allowing it to fluidly transition between actions such as planning, searching, and editing. This design mirrors the way engineers backtrack and adjust their approach dynamically, ensuring the agent can revise its course when faced with new challenges or information, and points towards the direction of more general, open-ended systems
+(Wang et al.,
+2023
+; Ma et al.,
+2024a
+; Lu et al.,
+2024b
+; Faldor et al.,
+2024
+; Hu et al.,
+2024
+; Lu et al.,
+2024a
+)
+.
+Iterative Learning through Feedback
+: Effective engineering relies heavily on continuous testing and refinement. To replicate this, SWE-Search integrates a Monte Carlo Tree Search (MCTS)
+(Silver et al.,
+2016b
+)
+planning module paired with a Value Agent. The MCTS module balances exploration and exploitation to guide the agent through complex solution spaces. The Value Agent augments this process by providing both utility estimates and qualitative feedback, allowing the agent to iteratively improve its decision-making based on past experiences, similar to how engineers refine their work through feedback and debugging.
+Collaborative Decision-Making
+: Complex problems often benefit from diverse perspectives
+(Khan et al.,
+2024
+; Amayuelas et al.,
+2024
+; Du et al.,
+2023
+; Zhang et al.,
+2024c
+)
+. In SWE-Search, once a set of potential solutions is generated, the Discriminator Agent facilitates a multi-agent debate. Each agent advocates for different solutions by presenting arguments, which are critically evaluated by a judge agent. This process mirrors real-world engineering collaboration, where teams deliberate to refine and select the most robust solutions.
+The architecture of SWE-Search is designed to automate software engineering tasks through these adaptive, feedback-driven, and collaborative processes. The SWE-Agent serves as the system’s problem solver, operating in a dynamic environment where it can backtrack and adapt its actions as necessary. The MCTS Planning Module efficiently guides exploration and exploitation, ensuring that the agent balances the need for innovation with the need to focus on promising solutions. The Value Agent provides continual feedback, offering both quantitative assessments and qualitative insights, helping the agent refine its strategy iteratively. Finally, the Discriminator Agent ensures that the final decision is rigorously vetted through a multi-agent debate, simulating the collaborative decision-making processes commonly found in engineering teams.
+We evaluate SWE-Search on the SWE-bench benchmark, a comprehensive dataset from real-world open-source repositories. SWE-bench tests agents’ ability to resolve software issues by generating code patches that fix failing tests. SWE-Search demonstrates a
+23
+%
+percent
+23
+23\%
+relative performance improvement across five models compared to standard open-source agents, highlighting the effectiveness of strategic search and iterative self-evaluation. Through detailed analysis, we explore how performance scales with increased search depth and identify key factors that enhance self-assessment in software agents. Our work demonstrates the potential of MCTS and iterative learning to improve agent reasoning and planning in dynamic, complex domains like software engineering, introducing a new paradigm for autonomous software development.
+2
+Related Work
+Search methods
+Various search approaches have been applied to Large Language Models (LLMs) to facilitate System 2
+(Kahneman,
+2011
+; Saha et al.,
+2024
+; Pan et al.,
+2023
+; Bounsi et al.,
+2024
+)
+thinking in non-linear reasoning structures. A critical feature of these approaches is their ability to backtrack. Unlike greedy processes
+(Black,
+2005
+)
+, search algorithms explore multiple branches at each step, potentially escaping paths that lead to dead ends. These methods differ in their strategies for exploring and memorizing possible choices, and in their heuristics for switching between them. Breadth-first search
+(Moore,
+1959
+)
+maintains all possible search paths, incurring significant memory and computational costs. Depth-first search
+(Cormen et al.,
+2009
+)
+, in contrast, prioritizes the most promising path in a more greedy manner. When applied to LLMs, these methods demonstrate a trade-off between diversity and quality in text generation
+(Yao et al.,
+2023
+)
+. The A
+∗
+algorithm
+(Hart et al.,
+1968
+)
+combines aspects of breadth-first and greedy search to find optimal solutions using a predetermined evaluation function. In this work, we adopt Monte Carlo Tree Search (MCTS)
+(Silver et al.,
+2016b
+)
+, an advanced search algorithm that conducts statistical tree search without requiring dedicated evaluation heuristics for each state. MCTS has achieved impressive results in complex strategy games
+(Silver et al.,
+2016a
+)
+, protein folding
+(Jumper et al.,
+2021
+)
+, and algorithm discovery
+(Fawzi et al.,
+2022
+)
+.
+Software Agents
+Software agents are designed to perform autonomous actions within large codebases. Given a repository-level task, these agents typically locate relevant files and code segments before implementing necessary changes. We focus on the SWE-bench task
+(Jimenez et al.,
+2024
+)
+, which involves resolving real-world GitHub issues. Among the agents with disclosed technical details on SWE-bench,
+Yang et al. (
+2024b
+)
+introduced the concept of agent-computer interfaces with SWE-agent. OpenDevin
+(Wang et al.,
+2024b
+)
+presents a collection of community-driven agents, including CodeAct
+(Wang et al.,
+2024a
+)
+. The Agentless approach demonstrated competitive performance using a simple two-step process of localization and repair. AutoCodeRover
+(Zhang et al.,
+2024d
+)
+incorporated advanced code tools such as abstract syntax trees and spectrum-based fault localization. The Alibaba Lingma Agent
+(Ma et al.,
+2024b
+)
+introduced a search-based approach for repository exploration, followed by a structured editing phase. While effective, it constitutes a more hand-designed solution specifically designed to interface with the search functionality of their agent.
+3
+Methodology
+SWE-Search is a multi-agent system designed to tackle complex software engineering tasks by integrating dynamic planning, value estimation, and deliberative decision-making. The core motivation behind this method is to emulate the sophisticated, iterative workflows of human software engineers, where exploration, planning, and collaboration are crucial to solving intricate problems. By leveraging the strengths of Monte Carlo Tree Search (MCTS) for planning, a Value Agent for utility estimation and feedback, and a Discriminator Agent for final decision-making through debate, SWE-Search provides a comprehensive, adaptive framework capable of navigating and solving real-world software engineering challenges.
+SWE-Search consists of four primary components that work in synergy:
+SWE-Search Framework and Action Agent
+: Building on the moatless-tools framework
+(Örwall,
+2024
+)
+, SWE-Search operates in a dynamic code environment with a flexible state-space and a git-like commit tree structure. This design facilitates efficient backtracking to previous states, enabling the Action Agent to explore diverse solution trajectories. The adaptable state-space enhances the system’s ability to exploit the MCTS module effectively.
+Search Algorithm
+: The core of SWE-Search’s exploration strategy is based on a Monte Carlo Tree Search (MCTS) which uses a heuristic-based selection process similar to AlphaZero
+(Silver et al.,
+2016a
+)
+, specifically tailored for software engineering tasks. This modified MCTS algorithm effectively balances exploration and exploitation, helping the agent explore a diverse set of solutions and converge quickly on the most promising strategies.
+Value (Function) Agent
+: To approximate the utility of each observation, we employ an LLM-based value function, which in addition to outputting a value, also generates an explanation in natural language. This explanation can be leveraged to improve subsequent actions from parent nodes, enabling iterative self-improvement of the search process.
+Discriminator Agent
+: In the final stage of SWE-Search, the Discriminator Agent evaluates the solutions generated by the search process. Inspired by multi-agent debate frameworks
+Du et al. (
+2023
+); Khan et al. (
+2024
+); Amayuelas et al. (
+2024
+)
+, this agent engages in a structured debate, where multiple agents argue for or against the proposed solutions. The debate process not only surfaces diverse perspectives but also leads to a more rigorously justified final decision.
+This system architecture combines the strengths of dynamic action selection, strategic planning, and collaborative deliberation, creating a comprehensive tool capable of handling the complexity and iterative nature of software engineering tasks.
+3.1
+Problem Formulation
+Figure 1:
+SWE-Search Overview.
+Tree search.
+Each state is represented as a node from which the agent can expand from, and each corresponding action is presented as an edge.
+Evaluation.
+Uses all relevant context including trajectory information, file context, and executed tests, to provide a quantitative value estimation and qualitative explanation in natural language.
+Expansion.
+Nodes can be expanded using value function feedback from future actions.
+The task of the SWE agent can be formalized as a tuple
+ℳ
+=
+(
+𝒮
+,
+𝒞
+,
+𝒜
+,
+𝒱
+,
+𝒫
+,
+p
+0
+,
+ρ
+)
+ℳ
+𝒮
+𝒞
+𝒜
+𝒱
+𝒫
+subscript
+𝑝
+0
+𝜌
+\mathcal{M}=(\mathcal{S},\mathcal{C},\mathcal{A},\mathcal{V},\mathcal{P},p_{0},\rho)
+. Here,
+𝒮
+𝒮
+\mathcal{S}
+represents the state space, encompassing all possible states such as the current context of the files the agent is working on and the overall status of the codebase. The context space, denoted as
+𝒞
+𝒞
+\mathcal{C}
+, includes metadata about the repository and the initial problem description. The value function
+𝒱
+𝒱
+\mathcal{V}
+assigns a utility score to each state-action pair
+O
+​
+(
+a
+,
+t
+)
+𝑂
+𝑎
+𝑡
+O(a,t)
+, guiding the agent’s decisions.
+The environment’s dynamics are defined by a context-dependent transition function
+𝒫
+:
+𝒮
+×
+𝒜
+×
+𝒞
+→
+Δ
+​
+(
+𝒮
+)
+:
+𝒫
+→
+𝒮
+𝒜
+𝒞
+Δ
+𝒮
+\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{C}\rightarrow\Delta(\mathcal{S})
+, which models the evolution of the repository’s state after each action. The initial state distribution,
+p
+0
+:
+𝒞
+→
+Δ
+​
+(
+𝒮
+)
+:
+subscript
+𝑝
+0
+→
+𝒞
+Δ
+���
+p_{0}:\mathcal{C}\rightarrow\Delta(\mathcal{S})
+, specifies how the initial state depends on the given context, while
+ρ
+∈
+Δ
+​
+(
+𝒞
+)
+𝜌
+Δ
+𝒞
+\rho\in\Delta(\mathcal{C})
+defines the distribution over contexts.
+Given an initial context
+c
+∼
+ρ
+similar-to
+𝑐
+𝜌
+c\sim\rho
+and an initial state
+s
+0
+∼
+p
+0
+(
+⋅
+∣
+c
+)
+s_{0}\sim p_{0}(\cdot\mid c)
+, the SWE agent executes its policy
+π
+:
+𝒮
+×
+𝒞
+→
+Δ
+​
+(
+𝒜
+)
+:
+𝜋
+→
+𝒮
+𝒞
+Δ
+𝒜
+\pi:\mathcal{S}\times\mathcal{C}\rightarrow\Delta(\mathcal{A})
+, which selects actions based on the current state and context. At each time step
+t
+𝑡
+t
+, the agent takes an action
+a
+t
+∼
+π
+​
+(
+s
+t
+,
+c
+)
+similar-to
+subscript
+𝑎
+𝑡
+𝜋
+subscript
+𝑠
+𝑡
+𝑐
+a_{t}\sim\pi(s_{t},c)
+and receives a corresponding reward
+ℛ
+​
+(
+s
+t
+,
+a
+t
+,
+c
+)
+ℛ
+subscript
+𝑠
+𝑡
+subscript
+𝑎
+𝑡
+𝑐
+\mathcal{R}(s_{t},a_{t},c)
+. The environment then transitions to a new state
+s
+t
++
+1
+∼
+𝒫
+(
+⋅
+∣
+s
+t
+,
+a
+t
+,
+c
+)
+s_{t+1}\sim\mathcal{P}(\cdot\mid s_{t},a_{t},c)
+, and the agent continues to observe this updated state. Over time, this process generates a trajectory
+τ
+:=
+{
+s
+t
+,
+a
+t
+,
+r
+t
+}
+t
+=
+0
+T
+assign
+𝜏
+superscript
+subscript
+subscript
+𝑠
+𝑡
+subscript
+𝑎
+𝑡
+subscript
+𝑟
+𝑡
+𝑡
+0
+𝑇
+\tau:=\{s_{t},a_{t},r_{t}\}_{t=0}^{T}
+as the agent interacts with the environment.
+The agent’s objective is to maximize the cumulative reward over the trajectory, which is captured by the value function
+v
+​
+(
+s
+t
+,
+a
+t
+,
+{
+s
+i
+}
+i
+=
+0
+t
+−
+1
+,
+{
+a
+i
+}
+i
+=
+0
+t
+−
+1
+)
+𝑣
+subscript
+𝑠
+𝑡
+subscript
+𝑎
+𝑡
+superscript
+subscript
+subscript
+𝑠
+𝑖
+𝑖
+0
+𝑡
+1
+superscript
+subscript
+subscript
+𝑎
+𝑖
+𝑖
+0
+𝑡
+1
+v(s_{t},a_{t},\{s_{i}\}_{i=0}^{t-1},\{a_{i}\}_{i=0}^{t-1})
+. This value function depends not only on the current state and action but also on the history of previous states and actions, which deviates from the assumptions of a Markovian process. Formally, the agent seeks to maximize the expected cumulative reward, defined as:
+max
+π
+⁡
+V
+T
+​
+(
+ρ
+)
+=
+max
+π
+⁡
+𝔼
+​
+τ
+​
+[
+∑
+t
+=
+0
+T
+ℛ
+​
+(
+s
+t
+,
+a
+t
+,
+c
+)
+∣
+c
+∼
+ρ
+;
+π
+]
+subscript
+𝜋
+superscript
+𝑉
+𝑇
+𝜌
+subscript
+𝜋
+𝔼
+𝜏
+delimited-[]
+similar-to
+conditional
+superscript
+subscript
+𝑡
+0
+𝑇
+ℛ
+subscript
+𝑠
+𝑡
+subscript
+𝑎
+𝑡
+𝑐
+𝑐
+𝜌
+𝜋
+\max_{\pi}V^{T}(\rho)=\max_{\pi}\mathbb{E}{\tau}\left[\sum_{t=0}^{T}\mathcal{R}(s_{t},a_{t},c)\mid c\sim\rho;\pi\right]
+.
+This optimization captures the agent’s (in-context) process, as it adjusts its policy
+π
+𝜋
+\pi
+to achieve the highest expected return across multiple trajectories, considering both current and historical information.
+3.2
+SWE-Search Framework and Action Agent
+The SWE-Search Action Agent builds on the moatless-tools framework
+(Örwall,
+2024
+)
+. Its action space,
+𝒜
+𝒜
+\mathcal{A}
+, is organized as a two-tier hierarchy, comprising both action types and their corresponding specific actions. Formally, this can be expressed as
+𝒜
+=
+(
+t
+,
+a
+)
+∣
+t
+∈
+𝒯
+,
+a
+∈
+𝒜
+t
+formulae-sequence
+𝒜
+conditional
+𝑡
+𝑎
+𝑡
+𝒯
+𝑎
+subscript
+𝒜
+𝑡
+\mathcal{A}={(t,a)\mid t\in\mathcal{T},a\in\mathcal{A}_{t}}
+, where
+𝒯
+𝒯
+\mathcal{T}
+represents the set of action types (e.g.,
+Search
+,
+Plan
+,
+Edit
+), and
+𝒜
+t
+subscript
+𝒜
+𝑡
+\mathcal{A}_{t}
+is the set of possible actions corresponding to each type
+t
+𝑡
+t
+. These actions range from tool invocations and code modifications to the generation of structured text. To enhance the agent’s effectiveness in search-driven tasks, we introduced the following modifications:
+One key modification we implemented is the expansion of the
+Plan
+state, allowing it to transition flexibly to any other state, rather than being limited to transitioning only to
+Edit
+. This change is motivated by the need to enable more dynamic and adaptive problem-solving behaviors within the agent. In the context of software engineering, rigid state transitions can be overly restrictive, forcing the agent into predetermined pathways that may not always align with the complexities of real-world scenarios. For instance, during code modification tasks, an agent might recognize mid-process that further planning, additional searches, or different types of analysis are necessary before proceeding with edits. Restricting transitions only to editing would artificially constrain the agent, potentially leading it to suboptimal actions or causing it to become stuck in unproductive loops. By allowing transitions to any state, we empower the agent to adapt to new information as it arises (
+Fig.
+2
+), exploring a wider variety of trajectories. This enhanced flexibility reflects the iterative and often non-linear nature of real software engineering workflows, where engineers frequently revisit planning, testing, and research phases before committing to edits.
+Second, the agent is empowered to execute any tests within the codebase at its discretion, as well as to create and implement new tests. The results of these tests are incorporated into both the value function and the agent’s subsequent decision-making process. It is crucial to highlight that the tests required to resolve a given instance (i.e., fail-to-pass tests) are not explicitly revealed to the agent. However, the agent can leverage any pre-existing tests within the repository, simulating the behavior of a real-world software engineer.
+1
+1
+1
+This approach aligns with the practices of other SWE agents, and has been validated by the authors of SWE-bench, who confirmed its legitimacy as long as the fail-to-pass tests remain concealed from the model.
+3.3
+Value (Function) Agent
+The role of the Value Agent extends beyond simply estimating the expected utility of a given state-action pair
+O
+n
+​
+(
+s
+n
+,
+a
+n
+)
+subscript
+𝑂
+𝑛
+subscript
+𝑠
+𝑛
+subscript
+𝑎
+𝑛
+O_{n}(s_{n},a_{n})
+. In addition to calculating the value
+v
+n
+subscript
+𝑣
+𝑛
+v_{n}
+, the Value Agent generates a written explanation, denoted as
+ε
+𝜀
+\varepsilon
+. This explanation serves a dual purpose: it provides transparency into the decision-making process and functions as feedback for the Action Agent, which can leverage this explanation when re-expanding from the parent node of
+O
+n
+subscript
+𝑂
+𝑛
+O_{n}
+(see
+Figure
+1
+,
+hindsight feedback
+). This approach enables the system to iteratively refine its decision-making process, mirroring how a human software engineer continuously re-evaluates their approach based on new information to improve their problem-solving strategy.
+The input to the value function consists of all state-action pairs up to and including the current state being evaluated, alongside specific instructions on how to assess the state. This allows the Value Agent to contextualize the decision within the trajectory, accounting for the sequence of actions and states leading up to the present. The final output of the value function can be formalized as:
+(
+v
+t
+,
+ε
+t
+)
+=
+V
+​
+(
+s
+t
+,
+a
+t
+,
+{
+s
+i
+}
+i
+=
+0
+​
+…
+​
+t
+−
+1
+,
+{
+a
+i
+}
+i
+=
+0
+​
+…
+​
+t
+−
+1
+)
+subscript
+𝑣
+𝑡
+subscript
+𝜀
+𝑡
+𝑉
+subscript
+𝑠
+𝑡
+subscript
+𝑎
+𝑡
+subscript
+subscript
+𝑠
+𝑖
+𝑖
+0
+…
+𝑡
+1
+subscript
+subscript
+𝑎
+𝑖
+𝑖
+0
+…
+𝑡
+1
+(v_{t},\varepsilon_{t})=V(s_{t},a_{t},\{s_{i}\}_{i=0{\ldots}t-1},\{a_{i}\}_{i=0{\ldots}t-1})
+(1)
+Here,
+v
+t
+subscript
+𝑣
+𝑡
+v_{t}
+represents the expected utility of the current state-action pair, while
+ε
+t
+subscript
+𝜀
+𝑡
+\varepsilon_{t}
+is the accompanying explanation.
+In practice, the Value Agent is tasked with analyzing the entire trajectory leading up to the current state-action pair, providing not only the required utility estimate
+v
+t
+subscript
+𝑣
+𝑡
+v_{t}
+, but also a detailed explanation
+ε
+t
+subscript
+𝜀
+𝑡
+\varepsilon_{t}
+. This explanation is critical for the agent’s overall performance, as it offers insight into the reasoning behind utility estimates, which in turn informs the Action Agent’s future decisions. We have observed that one of the key factors driving the effectiveness of the Value Agent lies in the clarity and specificity of these explanations. A well-articulated explanation can illuminate the strengths and limitations of different state types (e.g.,
+Search
+,
+Edit
+,
+Plan
+), helping the Action Agent better understand which types of states are more promising or risky to pursue.
+By providing detailed feedback on the potential utility of different actions and contextualizing them within the broader trajectory, the Value Agent enables more informed and strategic decision-making by the Action Agent. This integration of both quantitative and qualitative feedback leads to improved performance and more adaptive behavior throughout the task (
+Fig.
+4
+a
+).
+3.4
+Search Algorithm
+Our search tree is structured with nodes representing states
+𝒮
+t
+subscript
+𝒮
+𝑡
+\mathcal{S}_{t}
+and edges representing actions
+𝒜
+t
+subscript
+𝒜
+𝑡
+\mathcal{A}_{t}
+. The search algorithm employed is a modified Monte Carlo Tree Search (MCTS), specifically adapted for the tasks of the SWE-Agent. Unlike prior approaches for web agents that utilize language models in the selection process
+Koh et al. (
+2024
+); Zhang et al. (
+2024b
+)
+, we deliberately choose not to rely on language models for node selection. Instead, we adopt a more straightforward heuristic-based selection function, similar to the approach used in AlphaZero
+Silver et al. (
+2016a
+;
+2018
+)
+. This decision is driven by the need for interpretability, efficiency, and the focus on tasks where heuristic-based exploration suffices to guide the agent effectively through complex software engineering environments.
+At the core of our algorithm is a modified Upper Confidence Bound for Trees (UCT) selection criterion
+Kocsis & Szepesvári (
+2006
+)
+, which determines the next node to expand. This criterion balances exploitation of known high-reward actions with exploration of less-visited states. We introduce additional terms to encourage strategic exploration early in the search process, and to penalize over-exploration at later stages when convergence on the optimal solution is desired. The modified UCT function is expressed as:
+U
+​
+C
+​
+T
+​
+(
+s
+,
+a
+)
+=
+e
+​
+x
+​
+p
+​
+l
+​
+o
+​
+i
+​
+t
+​
+a
+​
+t
+​
+i
+​
+o
+​
+n
++
+e
+​
+x
+​
+p
+​
+l
+​
+o
+​
+r
+​
+a
+​
+t
+​
+i
+​
+o
+​
+n
++
+e
+​
+a
+​
+r
+​
+l
+​
+y
+​
+_
+​
+d
+​
+e
+​
+p
+​
+t
+​
+h
+​
+_
+​
+b
+​
+o
+​
+n
+​
+u
+​
+s
+−
+l
+​
+a
+​
+t
+​
+e
+​
+_
+​
+d
+​
+e
+​
+p
+​
+t
+​
+h
+​
+_
+​
+p
+​
+e
+​
+n
+​
+a
+​
+l
+​
+t
+​
+y
+𝑈
+𝐶
+𝑇
+𝑠
+𝑎
+𝑒
+𝑥
+𝑝
+𝑙
+𝑜
+𝑖
+𝑡
+𝑎
+𝑡
+𝑖
+𝑜
+𝑛
+𝑒
+𝑥
+𝑝
+𝑙
+𝑜
+𝑟
+𝑎
+𝑡
+𝑖
+𝑜
+𝑛
+𝑒
+𝑎
+𝑟
+𝑙
+𝑦
+_
+𝑑
+𝑒
+𝑝
+𝑡
+ℎ
+_
+𝑏
+𝑜
+𝑛
+𝑢
+𝑠
+𝑙
+𝑎
+𝑡
+𝑒
+_
+𝑑
+𝑒
+𝑝
+𝑡
+ℎ
+_
+𝑝
+𝑒
+𝑛
+𝑎
+𝑙
+𝑡
+𝑦
+UCT(s,a)=exploitation+exploration+early\_depth\_bonus-late\_depth\_penalty
+(2)
+This can be expressed more formally as:
+U
+​
+C
+​
+T
+​
+(
+s
+,
+a
+)
+=
+V
+​
+(
+s
+,
+a
+)
++
+C
+​
+ln
+⁡
+N
+​
+(
+s
+)
+N
+​
+(
+s
+,
+a
+)
++
+α
+​
+e
+−
+β
+​
+(
+d
+−
+1
+)
+−
+γ
+​
+d
+𝑈
+𝐶
+𝑇
+𝑠
+𝑎
+𝑉
+𝑠
+𝑎
+𝐶
+𝑁
+𝑠
+𝑁
+𝑠
+𝑎
+𝛼
+superscript
+𝑒
+𝛽
+𝑑
+1
+𝛾
+𝑑
+UCT(s,a)=V(s,a)+C\sqrt{\frac{\ln N(s)}{N(s,a)}}+\alpha e^{-\beta(d-1)}-\gamma\sqrt{d}
+(3)
+V
+​
+(
+s
+,
+a
+)
+𝑉
+𝑠
+𝑎
+V(s,a)
+is the value estimate of the state-action pair
+,
+N
+​
+(
+s
+,
+a
+)
+𝑁
+𝑠
+𝑎
+N(s,a)
+is the number of times the state-action pair
+(
+s
+,
+a
+)
+𝑠
+𝑎
+(s,a)
+has been visited,
+N
+​
+(
+s
+)
+𝑁
+𝑠
+N(s)
+is the visit count of state
+s
+𝑠
+s
+,
+d
+𝑑
+d
+is the depth of the node in the search tree, and
+C
+𝐶
+C
+,
+α
+𝛼
+\alpha
+,
+β
+𝛽
+\beta
+, and
+γ
+𝛾
+\gamma
+are constants that control the balance between exploration, exploitation, and depth-dependent rewards and penalties.
+This formulation is inspired by the way software engineers explore potential solutions to a task. In practice, an engineer’s search process can be broken down into the following key phases, which our algorithm mirrors:
+Early Exploration
+: Initially, an engineer explores a wide variety of potential approaches to fully understand the problem and identify promising strategies. This is encouraged in our algorithm by the
+e
+​
+a
+​
+r
+​
+l
+​
+y
+​
+_
+​
+d
+​
+e
+​
+p
+​
+t
+​
+h
+​
+_
+​
+b
+​
+o
+​
+n
+​
+u
+​
+s
+𝑒
+𝑎
+𝑟
+𝑙
+𝑦
+_
+𝑑
+𝑒
+𝑝
+𝑡
+ℎ
+_
+𝑏
+𝑜
+𝑛
+𝑢
+𝑠
+early\_depth\_bonus
+, represented by the term
+α
+​
+e
+−
+β
+​
+(
+d
+−
+1
+)
+𝛼
+superscript
+𝑒
+𝛽
+𝑑
+1
+\alpha e^{-\beta(d-1)}
+, which rewards exploration at shallow depths, simulating the early phases of wide exploration.
+Convergence and Exploitation
+: As the engineer gains more information and narrows down the options, the focus shifts to exploiting the most effective solution paths. This transition is handled by the standard UCT exploitation term
+V
+​
+(
+s
+,
+a
+)
+𝑉
+𝑠
+𝑎
+V(s,a)
+and is further reinforced by the
+l
+​
+a
+​
+t
+​
+e
+​
+_
+​
+d
+​
+e
+​
+p
+​
+t
+​
+h
+​
+_
+​
+p
+​
+e
+​
+n
+​
+a
+​
+l
+​
+t
+​
+y
+𝑙
+𝑎
+𝑡
+𝑒
+_
+𝑑
+𝑒
+𝑝
+𝑡
+ℎ
+_
+𝑝
+𝑒
+𝑛
+𝑎
+𝑙
+𝑡
+𝑦
+late\_depth\_penalty
+(
+−
+γ
+​
+d
+𝛾
+𝑑
+-\gamma\sqrt{d}
+), which discourages over-exploration as the agent delves deeper into the search tree.
+Quick Abandonment of Poor Strategies
+: Software engineers are also adept at abandoning poor strategies when new information indicates that a particular approach is not viable. We capture this behavior by implementing a simple heuristic rule that abandons nodes associated with consecutive low rewards, ensuring that the agent does not waste resources on unproductive trajectories.
+At each step, the node with the highest UCT value is selected for expansion, formalized as:
+s
+∗
+=
+arg
+​
+max
+(
+s
+,
+a
+)
+⁡
+U
+​
+C
+​
+T
+​
+(
+s
+,
+a
+)
+superscript
+𝑠
+subscript
+arg
+max
+𝑠
+𝑎
+𝑈
+𝐶
+𝑇
+𝑠
+𝑎
+s^{*}=\operatorname*{arg\,max}_{(s,a)}UCT(s,a)
+(4)
+This approach effectively mimics the decision-making process of a software engineer, who balances exploration of potential strategies with a focus on converging towards the optimal solution, while remaining flexible enough to backtrack when necessary. By incorporating heuristic feedback and depth-based adjustments, the algorithm avoids getting stuck in unproductive paths and enhances the agent’s ability to identify high-reward strategies with minimal computational overhead
+Appendix
+6
+.
+3.4.1
+Discriminator Agent
+The final stage of SWE-Search involves the Discriminator Agent, whose role is to evaluate the candidate solutions generated by the search process and select the one most likely to resolve the issue at hand. This module accepts up to five final solutions produced by the search and engages in a multi-agent debate to determine the most promising option. Drawing inspiration from recent work on persuasive multi-agent debates
+(Khan et al.,
+2024
+; Amayuelas et al.,
+2024
+)
+, the Discriminator leverages the collective reasoning of multiple agents to ensure a more robust final selection. Configuration and hyperparameter details can be found in
+Table
+2
+.
+In this stage, agents are presented with the original problem statement and candidate solutions. They engage in a structured debate to determine the most effective solution, supporting their choices with logical reasoning and evidence from the search process. This debate encourages a thorough exploration of trade-offs between solutions, potentially uncovering strengths or weaknesses not evident during individual searches. Finally, a judge agent evaluates the arguments and selects the solution deemed most likely to resolve the issue. This process simulates the collaborative decision-making in software engineering teams, where diverse perspectives lead to a more thorough evaluation of candidate solutions, ultimately increasing the likelihood of identifying the most optimal outcome.
+The discriminator process not only enhances the robustness of the final solution but also adds transparency, as the reasoning behind the choice is clearly articulated and evaluated. This ensures that the selected solution is well-reasoned and thoroughly vetted before implementation.
+Figure 2:
+Hindsight feedback error correction.
+Instance sympy__sympy-15678, SWE-Search with Qwen2.5-72B-Instruct. Initially, the Action Agent performs edits and runs tests, which pass. It prematurely concludes the search. Without actually knowing the proposed solution does not resolve the issue, the Value Agent identifies potentially missed tests and assigns a low reward. Upon re-expansion using the Value Agent’s feedback, new tests fail, prompting the Action Agent to make additional edits, which result in a preferred solution which ultimately resolves the issue.
+4
+Experiments
+Benchmark
+For our experiments, we utilize SWE-bench Lite, a curated subset of the official SWE-bench, containing 300 instances. This dataset is specifically designed to be self-contained and focuses primarily on evaluating functional bug fixes, providing a controlled environment to assess the performance of our system.
+Evaluation Metrics
+We use two metrics: resolve rate (
+Pass@1
+) and
+Pass@5
+. Resolve rate is the percentage of issues successfully resolved, measuring overall effectiveness. Pass@5 is the percentage of issues where a correct solution is found within five attempts. This allows us to assess the efficiency of the search in identifying successful bug fixes within a limited number of iterations.
+Baselines
+Software agents leverage diverse tools, architectures, and models, leading to variability in their performance on subsets of the SWE-bench Lite dataset
+(Zhang et al.,
+2024a
+)
+. For comparison, we build upon the moatless-tools framework
+(Örwall,
+2024
+)
+, a high-performing open-source agent commonly used in research settings
+(Chowdhury et al.,
+2024
+)
+. To isolate the impact of our search approach, we adapt moatless-tools as our baseline, referred to as moatless-adapted. This allows us to fairly compare the performance of SWE-Search against moatless-adapted across various models, including two closed-source models (GPT-4o, GPT-4o-mini) and three open-source models (Qwen2.5-72B-Instruct
+(Yang et al.,
+2024a
+)
+, Llama-3.1-70B-Instruct
+(Dubey et al.,
+2024
+)
+, and DeepSeek-V2.5
+(DeepSeek-AI et al.,
+2024
+)
+). We also reference official moatless-tools GPT-4o results on SWE-bench Lite to ensure a fair and consistent comparison.
+Implementation Details
+For consistency, we use identical prompts across all models. In SWE-Search, we limit each node to a maximum of three expansions and cap the total search iterations at 100. Further details on model hyperparameters can be found in
+Appendix,
+2
+.
+Table 1:
+Resolve Rate Comparison, SWE-bench Lite
+Model
+Moatless-v1
+Moatless-adapted
+SWE-Search
+%
+Δ
+Δ
+\Delta
+GPT-4o
+24.3
+25.7
+31.0
++17
+GPT-4o-mini
+–
+13.0
+17.0
++24
+Qwen-2.5-72b-Instruct
+–
+18.0
+24.7
++27
+Deepseek-V2.5
+–
+16.3
+21.0
++22
+Llama-3.1-70b-Instruct
+–
+13.6
+17.7
++23
+Mean %
+Δ
+Δ
+\Delta
++23
+4.1
+Experimental Results
+4.1.1
+SWE-Search Surpasses all Corresponding Base Agents and Enables Smaller, Open Source Models to Approach GPT-4o
+On average, SWE-Search outperforms the baseline agent across all five models, achieving a 23% relative improvement
+(Table
+1
+)
+. Notably, SWE-Search with Qwen-2.5-72B-Instruct exceeds the performance of GPT-4o using the original Moatless-v1 framework, and closely matches its performance when compared with the Moatless-adapted agent, with only a slight difference (
+Δ
+=
+−
+1
+%
+Δ
+percent
+1
+\Delta=-1\%
+). Interestingly, all five models demonstrate significant improvement when utilizing the proposed approach, with consistent gains across different models.
+4.1.2
+Search Enables Agents to Make Better Use of More Flexibility
+To prevent goal divergence, most agents, including moatless-tools, rely on strict transition rules, where state transitions follow predetermined sequences (e.g., Search
+→
+→
+\rightarrow
+Identify, Plan
+→
+→
+\rightarrow
+Edit). In moatless-adapted, we introduce a more flexible transition logic that allows a Plan state to transition into any other state type. This added flexibility has both advantages and drawbacks. On the positive side, it enables the agent to autonomously correct its trajectory without external feedback, particularly when the necessary adjustments span only a limited portion of the task. However, this increased flexibility also introduces the risk of the agent becoming trapped in infinite loops. Without a high-level control mechanism to detect and mitigate these situations, the agent may fail to recover from such loops. This trade-off is evident in the modest performance difference between Moatless-v1 and moatless-adapted, with a slight performance improvement of only 1.4% (
+Table
+1
+).
+4.1.3
+Impact of Hindsight Feedback on Agent Performance
+One key advantage of utilizing LLMs as general value functions is their dual ability to provide both quantitative value estimates and qualitative assessments in natural language. These qualitative insights can significantly enhance the agent’s action generation and search process by offering detailed feedback on potential errors or overlooked aspects of the task. In practice, feedback was also crucial in eliciting diversity in the actions taken by the agent, as without it, the agent would often take very similar actions when re-expanding from a parent node.
+As shown in
+Figure
+2
+, this mechanism plays a critical role in improving the agent’s performance. During the initial expansion, the agent prematurely concludes that the task is complete. However, the value function correctly identifies gaps in the test coverage, specifically in addressing potential corner cases, and assigns a low reward. This feedback prompts the agent to re-expand the parent state, leading to the introduction of new tests, which subsequently fail. The agent then performs a series of edits (summarized in the figure for brevity), ultimately resolving the task correctly. Empirically, we observe that the instances unresolved by moatless-adapted but successfully solved by SWE-Search are often attributed to this search-and-feedback loop, where iterative feedback drives the agent toward a correct solution.
+4.2
+Importance of Comprehensive State Information for Value Function Performance
+Model
+Pass@1
+Pass@5
+GPT-4o
+31.0
+34.0
+GPT-4o-mini
+17.0
+22.3
+Qwen-2.5-72b-Instruct
+24.7
+25.7
+Deepseek-V2.5
+21.0
+23.3
+Llama-3.1-70b-Instruct
+21.0
+22.3
+Figure 3:
+SWE-bench SWE-Search results
+The effectiveness of SWE-Search hinges on the value function’s ability to accurately differentiate between desirable and undesirable states, and to provide actionable feedback that drives improvement. However, our experiments revealed that the value function sometimes failed to recognize critical decision points in the search tree. It frequently misinterpreted the purpose of certain actions, leading to the undervaluation of effective strategies by assigning low rewards. As shown in
+Figure
+4
+a
+, before the introduction of state-specific value prompts, the agent consistently assigned low rewards even when the Action Agent correctly identified the need for additional context, such as locating relevant files. This issue persisted despite the agent successfully identifying the files later. By implementing state-specific prompts across core state clusters (Searching, Planning, Editing), the value function became significantly more adept at interpreting the intent behind actions and evaluating their outcomes within each state. For further details on experiments distinguishing between effective and ineffective states, refer to
+Appendix
+8
+.
+Figure 4:
+(a) Importance of state-specific value prompts.
+On the left and right are the respective Value Agents’ outputs with and without state-specific prompts. While the action in both cases is effective in finding the right file, the non-state-specific scenario does not recognize this and assigns a low reward. On the contrary, the state-specific prompt correctly assigns a high reward to this state.
+(b) Performance scaling with search depth across different language models.
+The graph shows the number of issues resolved as a function of the number of transitions (search iterations) for all models used.
+Scaling SWE agents with Inference-time Compute
+The success of large language models (LLMs) has traditionally been attributed to the expansion of training data and model size, i.e., training-time compute
+(Wei et al.,
+2022
+; Chung et al.,
+2022
+)
+. Recently, researchers have started exploring how different methods scale with inference-time
+(OpenAI,
+2024
+; Snell et al.,
+2024
+; Dubey et al.,
+2024
+)
+. Here, we study the performance of software engineering agents through increased inference-time compute. As shown in
+Figure
+4
+b
+, increasing search iterations leads to a consistent rise in the number of resolved issues. To ensure experimental feasibility across the 300 instances in the SWE-bench Lite dataset, we applied conservative parameters (maximum iterations
+=
+100
+absent
+100
+=100
+, maximum expansions per node
+=
+3
+absent
+3
+=3
+). Approaches like SWE-Search enable the allocation of greater resources to specific challenges, such as addressing critical software vulnerabilities
+(Rigaki et al.,
+2024
+; Fang et al.,
+2024
+)
+, offering a scalable solution to complex tasks.
+Figure 5:
+(a) Value Function vs. Discriminator Comparison.
+Comparison of value function vs. discriminator ability to discern the final solution that resolved the issue when there is one. The discriminator performs better across all models except GPT-4o-mini. DeepSeek-V2.5 had the smallest disparity between the two methods, suggesting an ability to act as a well-calibrated value function.
+(b) Model-Specific Issue Resolution.
+Venn diagram of resolved issues by model. Each model can solve a handful of unique instances.
+Convergence of Value Function and Discriminator to Right Solution
+The search process can yield multiple proposed solutions. Ideally, the mean trajectory value of the the proposed solution that resolves the issue will always be the highest, which would yields the ideal performance of the agent
+(Table
+3
+)
+. In practice, the value function successfully converged on the correct solution 73% of the time on average across the five models. The discriminator module performed even better, increasing the proportion of correct solutions selected to 84%. While in typical large action spaces, Monte Carlo Tree Search (MCTS) is run for thousands of iterations
+(Silver et al.,
+2016b
+)
+, the value function’s success rate remains impressive given the computational constraints. However, SWE-Search could further benefit from enhanced methods for identifying the correct solutions more consistently, allowing it to fully reach its potential.
+Different Models can Resolve Vastly Different Issue Subsets
+When comparing the resolved instances across the five models, we observed significant diversity in the subsets of issues each model successfully solved. As shown in
+Figure
+5
+, each model managed to resolve at least one unique instance. Notably, a surprising number of issues (33) were solved by other models but not by GPT-4o. This suggests that model diversity could play an important role, at least in the short term, in enhancing the performance of SWE-agents.
+5
+Discussion and Conclusion
+In this paper, we introduced SWE-Search, a general framework that integrates Monte Carlo Tree Search (MCTS) and qualitative feedback to enhance the performance of software engineering agents. The proposed approach demonstrated improvements over different baseline models, highlighting the potential of search-based methods in software engineering tasks.
+One of the key advantages of search-based approaches, as demonstrated in our work, is their ability to scale performance with increased inference-time compute. This flexibility allows the system to adapt to problems that require higher computational resources, such as discovering software vulnerabilities or even generating large codebases from scratch. Future research should focus on two main directions: (a) investigating how search agents scale with computational resources, and (b) expanding the application of software agent search to a broader range of complex use cases.
+Given that search techniques like MCTS closely resemble the problem-solving processes of human software engineers, we expect these methods to become increasingly prevalent in agent-driven systems. As the nature of software engineering tasks evolves, system architectures will need to become more fluid and adaptable, fully leveraging the potential of search-based techniques. This evolution will likely lead to the development of larger, more general agentic systems capable of tackling a wide array of software engineering challenges.
+References
+Amayuelas et al. (2024)
+Alfonso Amayuelas, Xianjun Yang, Antonis Antoniades, Wenyue Hua, Liangming Pan, and William Wang.
+Multiagent collaboration attack: Investigating adversarial attacks in large language model collaborations via debate, 2024.
+URL
+https://arxiv.org/abs/2406.14711
+.
+Black (2005)
+Paul E. Black.
+greedy algorithm, feb 2005.
+URL
+https://www.nist.gov/dads/HTML/greedyalgo.html
+.
+Accessed: TODAY.
+Bounsi et al. (2024)
+Wilfried Bounsi, Borja Ibarz, Andrew Dudzik, Jessica B. Hamrick, Larisa Markeeva, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković.
+Transformers meet neural algorithmic reasoners, 2024.
+URL
+https://arxiv.org/abs/2406.09308
+.
+Chowdhury et al. (2024)
+Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry.
+Introducing SWE-bench verified, August 2024.
+URL
+https://openai.com/research/introducing-swe-bench-verified
+.
+OpenAI Blog.
+Chung et al. (2022)
+Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.
+Scaling instruction-finetuned language models, 2022.
+URL
+https://arxiv.org/abs/2210.11416
+.
+Cormen et al. (2009)
+Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein.
+Introduction to Algorithms, Third Edition
+.
+The MIT Press, 3rd edition, 2009.
+ISBN 0262033844.
+DeepSeek-AI et al. (2024)
+DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang.
+Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024.
+URL
+https://arxiv.org/abs/2406.11931
+.
+Du et al. (2023)
+Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch.
+Improving factuality and reasoning in language models through multiagent debate, 2023.
+URL
+https://arxiv.org/abs/2305.14325
+.
+Dubey et al. (2024)
+Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov,
+Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier
+Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu,
+Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl
+Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James
+Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini
+Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta,
+Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vítor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao.
+The llama 3 herd of models, 2024.
+URL
+https://arxiv.org/abs/2407.21783
+.
+Faldor et al. (2024)
+Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune.
+Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code, 2024.
+URL
+https://arxiv.org/abs/2405.15568
+.
+Fang et al. (2024)
+Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang.
+Teams of llm agents can exploit zero-day vulnerabilities, 2024.
+URL
+https://arxiv.org/abs/2406.01637
+.
+Fawzi et al. (2022)
+A. Fawzi, M. Balog, A. Huang, T. Hubert, B. Romera-Paredes, M. Barekatain, A. Novikov, F. J. R. Ruiz, J. Schrittwieser, G. Swirszcz, D. Silver, D. Hassabis, and P. Kohli.
+Discovering faster matrix multiplication algorithms with reinforcement learning.
+Nature
+, 610(7930):47–53, 2022.
+doi:
+10.1038/s41586-022-05172-4
+.
+Hart et al. (1968)
+Peter E. Hart, Nils J. Nilsson, and Bertram Raphael.
+A formal basis for the heuristic determination of minimum cost paths.
+IEEE Trans. Syst. Sci. Cybern.
+, 4(2):100–107, 1968.
+doi:
+10.1109/TSSC.1968.300136
+.
+URL
+https://doi.org/10.1109/TSSC.1968.300136
+.
+Hu et al. (2024)
+Shengran Hu, Cong Lu, and Jeff Clune.
+Automated design of agentic systems, 2024.
+URL
+https://arxiv.org/abs/2408.08435
+.
+Jimenez et al. (2024)
+Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.
+Swe-bench: Can language models resolve real-world github issues?, 2024.
+URL
+https://arxiv.org/abs/2310.06770
+.
+Jumper et al. (2021)
+J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis.
+Highly accurate protein structure prediction with AlphaFold.
+Nature
+, 596(7873):583–589, 2021.
+doi:
+10.1038/s41586-021-03819-2
+.
+Kahneman (2011)
+Daniel Kahneman.
+Thinking, fast and slow
+.
+Farrar, Straus and Giroux, New York, NY, US, 2011.
+ISBN 978-0-374-27563-1.
+Khan et al. (2024)
+Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez.
+Debating with more persuasive llms leads to more truthful answers, 2024.
+URL
+https://arxiv.org/abs/2402.06782
+.
+Kocsis & Szepesvári (2006)
+Levente Kocsis and Csaba Szepesvári.
+Bandit based monte-carlo planning.
+In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou (eds.),
+Machine Learning: ECML 2006
+, pp.  282–293, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
+ISBN 978-3-540-46056-5.
+Koh et al. (2024)
+Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov.
+Tree search for language model agents, 2024.
+URL
+https://arxiv.org/abs/2407.01476
+.
+Li et al. (2022)
+Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.
+Competition-level code generation with alphacode.
+Science
+, 378(6624):1092–1097, 2022.
+doi:
+10.1126/science.abq1158
+.
+URL
+https://www.science.org/doi/abs/10.1126/science.abq1158
+.
+Lu et al. (2024a)
+Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha.
+The ai scientist: Towards fully automated open-ended scientific discovery, 2024a.
+URL
+https://arxiv.org/abs/2408.06292
+.
+Lu et al. (2024b)
+Cong Lu, Shengran Hu, and Jeff Clune.
+Intelligent go-explore: Standing on the shoulders of giant foundation models, 2024b.
+URL
+https://arxiv.org/abs/2405.15143
+.
+Ma et al. (2024a)
+Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
+Eureka: Human-level reward design via coding large language models, 2024a.
+URL
+https://arxiv.org/abs/2310.12931
+.
+Ma et al. (2024b)
+Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li.
+How to understand whole software repository?, 2024b.
+URL
+https://arxiv.org/abs/2406.01422
+.
+Moore (1959)
+E.F. Moore.
+The Shortest Path Through a Maze
+.
+Bell Telephone System. Technical publications. monograph. Bell Telephone System., 1959.
+URL
+https://books.google.com/books?id=IVZBHAAACAAJ
+.
+OpenAI (2024)
+OpenAI.
+OpenAI o1 System Card, September 2024.
+URL
+https://openai.com/research/o1-system-card
+.
+Online report.
+OpenAI et al. (2024)
+OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo
+Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan
+Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez,
+Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia
+Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.
+Gpt-4 technical report, 2024.
+URL
+https://arxiv.org/abs/2303.08774
+.
+Örwall (2024)
+Albert Örwall.
+Moatless tools, jun 2024.
+URL
+https://github.com/aorwall/moatless-tools
+.
+Accessed: 2024-07-16.
+Ouyang et al. (2022)
+Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.
+Training language models to follow instructions with human feedback, 2022.
+URL
+https://arxiv.org/abs/2203.02155
+.
+Pan et al. (2023)
+Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang.
+Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning, 2023.
+URL
+https://arxiv.org/abs/2305.12295
+.
+Rigaki et al. (2024)
+Maria Rigaki, Carlos Catania, and Sebastian Garcia.
+Hackphyr: A local fine-tuned llm agent for network security environments, 2024.
+URL
+https://arxiv.org/abs/2409.11276
+.
+Saha et al. (2024)
+Swarnadeep Saha, Archiki Prasad, Justin Chih-Yao Chen, Peter Hase, Elias Stengel-Eskin, and Mohit Bansal.
+System-1.x: Learning to balance fast and slow planning with language models, 2024.
+URL
+https://arxiv.org/abs/2407.14414
+.
+Silver et al. (2016a)
+David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.
+Mastering the game of go with deep neural networks and tree search.
+Nature
+, 529(7587):484–489, 1 2016a.
+ISSN 1476-4687.
+doi:
+10.1038/nature16961
+.
+URL
+https://doi.org/10.1038/nature16961
+.
+Silver et al. (2016b)
+David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.
+Mastering the game of go with deep neural networks and tree search.
+Nat.
+, 529(7587):484–489, 2016b.
+doi:
+10.1038/NATURE16961
+.
+URL
+https://doi.org/10.1038/nature16961
+.
+Silver et al. (2018)
+David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis.
+A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.
+Science
+, 362(6419):1140–1144, 2018.
+doi:
+10.1126/science.aar6404
+.
+URL
+https://www.science.org/doi/abs/10.1126/science.aar6404
+.
+Snell et al. (2024)
+Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
+Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024.
+URL
+https://arxiv.org/abs/2408.03314
+.
+Wang et al. (2023)
+Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
+Voyager: An open-ended embodied agent with large language models, 2023.
+URL
+https://arxiv.org/abs/2305.16291
+.
+Wang et al. (2024a)
+Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji.
+Executable code actions elicit better llm agents, 2024a.
+URL
+https://arxiv.org/abs/2402.01030
+.
+Wang et al. (2024b)
+Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig.
+Opendevin: An open platform for ai software developers as generalist agents, 2024b.
+URL
+https://arxiv.org/abs/2407.16741
+.
+Wei et al. (2022)
+Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.
+Emergent abilities of large language models.
+arXiv preprint arXiv:2206.07682
+, 2022.
+Xia et al. (2024)
+Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang.
+Agentless: Demystifying llm-based software engineering agents, 2024.
+URL
+https://arxiv.org/abs/2407.01489
+.
+Yang et al. (2024a)
+An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan.
+Qwen2 technical report, 2024a.
+URL
+https://arxiv.org/abs/2407.10671
+.
+Yang et al. (2024b)
+John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.
+Swe-agent: Agent-computer interfaces enable automated software engineering, 2024b.
+URL
+https://arxiv.org/abs/2405.15793
+.
+Yao et al. (2023)
+Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.
+Tree of thoughts: Deliberate problem solving with large language models.
+In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.),
+Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023
+, 2023.
+URL
+http://papers.nips.cc/paper_files/paper/2023/hash/271db9922b8d1f4dd7aaef84ed5ac703-Abstract-Conference.html
+.
+Zhang et al. (2024a)
+Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, Bo Pang, Yingbo Zhou, Shelby Heinecke, Silvio Savarese, Huan Wang, and Caiming Xiong.
+Diversity empowers intelligence: Integrating expertise of software engineering agents, 2024a.
+URL
+https://arxiv.org/abs/2408.07060
+.
+Zhang et al. (2024b)
+Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp.
+Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration, 2024b.
+URL
+https://arxiv.org/abs/2408.15978
+.
+Zhang et al. (2024c)
+Yiqun Zhang, Xiaocui Yang, Shi Feng, Daling Wang, Yifei Zhang, and Kaisong Song.
+Can llms beat humans in debating? a dynamic multi-agent framework for competitive debate, 2024c.
+URL
+https://arxiv.org/abs/2408.04472
+.
+Zhang et al. (2024d)
+Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury.
+Autocoderover: Autonomous program improvement, 2024d.
+URL
+https://arxiv.org/abs/2404.05427
+.
+Appendix A
+Reproducibility
+All models and data used in our work are publicly available. We additionally provide hyperparameter details in
+Appendix
+2
+. The code will be released as a public repository upon publication.
+Appendix B
+Additional Implementation Details
+Moatless-adapted is an extended version of the moatless-tools library with support for a tree structure, the ability to revert to earlier versions of the codebase, and the capability to run tests.
+The standard implementation of moatless-tools is based on a finite state machine structure where a state holds information about file context and properties set in the configuration or from previous states. It can then transition to a new state when an action is executed. The request that initiates the action is created by an LLM. This follows a linear structure where one state can transition to another state. In moatless-adapted, this model is extended so that a state can expand by using actions to create more states. The connections between states are then represented in a tree structure with nodes.
+Each state has a file context associated with it. This file context will be included in the prompt sent to an LLM. To limit the size of the prompt, files are divided into ”spans,” where a span could be, for example, a section of code (e.g., imports), a class, or a function. These are identified by span IDs. Thus, the LLM sees a limited part of the code at a time but can request more context by searching for or adding files and spans. The file context therefore changes over time, and a specific state of file context is linked to a specific state.
+In the standard implementation of moatless-tools, changes to the codebase are made linearly, and each change is saved directly to the file system. In moatless-adapted, however, there is a need to be able to revert to earlier states and thus return to a previous version of the codebase. To handle this, the code is stored in a git repository where each change is committed, and each state has a reference to a commit as well as the current patch of the diff from the initial commit that existed before starting. This way, one can go back to an earlier state by specifying the state ID, and the commit that was current at that time will be checked out.
+The test files present in the file context are run each time the Plan state is initiated, and the test results are provided to the state. The tests are then run in Docker images built via the SWE-bench library. To use this approach in a benchmark where a larger number of instances should be able to run simultaneously, a solution is used where these images are run as pods in a Kubernetes cluster. Moatless-tools communicates with the testbed by applying patches and running commands via an API. When a new instance starts, a pod is created which is then reset at each run, applying the current patch and running tests according to the test command specified in the SWE-bench library. It’s important to add here that the agent is not aware of the
+PASS_TO_PASS
+or
+FAIL_TO_PASS
+tests in the SWE-bench harness, but only knows how to run the tests. This corresponds to a real engineering environment where each project can have its own test commands.
+Appendix C
+MCTS Hyperparameters
+The Monte Carlo Tree Search (MCTS) algorithm used in this study employs several hyperparameters.
+Table 2:
+MCTS Hyperparameters
+Hyperparameter
+Description
+Default
+c_param
+UCT exploration parameter
+1.41
+max_expansions
+Max children per node
+5
+max_iterations
+Max MCTS iterations
+100
+provide_feedback
+Enable feedback
+True
+best_first
+Use best-first strategy
+True
+value_function_temperature
+Value function temperature
+0.2
+max_depth
+Max tree depth
+20
+UCT Score Calculation Parameters
+exploration_weight
+UCT exploration weight
+1.0
+depth_weight
+Depth penalty weight
+0.8
+depth_bonus_factor
+Depth bonus factor
+200.0
+high_value_threshold
+High-value node threshold
+55.0
+low_value_threshold
+Low-value node threshold
+50.0
+very_high_value_threshold
+Very high-value threshold
+75.0
+high_value_leaf_bonus_constant
+High-value leaf bonus
+20.0
+high_value_bad_children_bonus_constant
+High-value bad children bonus
+20.0
+high_value_child_penalty_constant
+High-value child penalty
+5.0
+Action Model Parameters
+action_model_temperature
+Action model temperature
+0.2
+Discriminator Parameters
+number_of_agents
+Number of Discriminator Agents
+5
+number_of_round
+Number of debate rounds
+3
+discriminator_temperature
+Discriminator temperature
+1.0
+These hyperparameters can be adjusted to fine-tune the MCTS algorithm’s performance for specific problem domains or computational constraints. The values listed here are the defaults as defined in the
+TreeSearchSettings
+class and the MCTS implementation.
+Appendix D
+Ability of MCTS to Escape Unproductive Loops vs. Baseline
+Figure 6:
+Avoiding Repetitive Actions, django__django__10914.
+We found that the base agent can often get stuck performing repetitive actions
+that do not bring it closer to solving the issue, and which commonly lead to unresolvable dead-ends. In this example, the base agent
+was stuck implementing wrong tests which continuously returned errors. In contrast, when this happens in
+SWE-Search, the Value Agent recognizes this, terminating these trajectories quickly,
+as happens in Node 73 (orange).
+Appendix E
+Model Instance Resolution Uniqueness
+To understand the complementary strengths of different models in resolving software issues, we analyzed how unique their resolved issue subsets where. Figure
+7
+illustrates the resolution patterns for each model across five of the codebases in SWE-bench-lite.
+Figure 7:
+Unique Issue Resolution Patterns Across Models and Libraries.
+Each column represents a different Python reposiroty, and each row within a column represents a specific issue. Colored blocks indicate successful resolution by the corresponding model (see legend). White spaces denote unresolved issues. This visualization highlights the diverse problem-solving capabilities of different models across various software domains, demonstrating that no single model dominates across all issues and libraries.
+Appendix F
+Ability of Value Function to Discern Successful Trajectories
+Before implementing SWE-Search, we conducted a general study across many models to evaluate the models’ ability to differentiate states which led to resolved vs. unresolved issues. Figure
+8
+shows the results of this study. We found that in general, models assigned higher rewards to states which eventually led to resolved issues. Of particular interest was the Deepseek model, which seemed to identify critical errors in trajectories effectively. This was also observed in the final agent (see Fig.
+5
+a).
+Figure 8:
+Average State Reward Comparison Across Models.
+This graph compares the average state rewards assigned by different language models for resolved (green) and unresolved (red) issues. Error bars indicate standard deviation. Most models consistently assign higher rewards to states leading to resolved issues, with the exception of the. The ’Average’ column represents the mean across all models, demonstrating a clear distinction between resolved and unresolved states.
+Appendix G
+Value Function Prompts
+◄
+Feeling
+lucky?
+Conversion
+report
+Report
+an issue
+View original
+on arXiv
+►

research/notes/241221139-training-software-engineering-agents-and-verifiers-with-swe-gym.md

@@ -0,0 +1,200 @@
+---
+title: '[2412.21139] Training Software Engineering Agents and Verifiers with SWE-Gym'
+id: 241221139-training-software-engineering-agents-and-verifiers-with-swe-gym
+tags:
+- deepread
+created: '2026-06-10T00:22:57.639430Z'
+source: https://arxiv.org/abs/2412.21139
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:22:57.639309Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2412.21139] Training Software Engineering Agents and Verifiers with SWE-Gym
+Computer Science > Software Engineering
+arXiv:2412.21139
+(cs)
+[Submitted on 30 Dec 2024 (
+v1
+), last revised 6 Jun 2025 (this version, v2)]
+Title:
+Training Software Engineering Agents and Verifiers with SWE-Gym
+Authors:
+Jiayi Pan
+,
+Xingyao Wang
+,
+Graham Neubig
+,
+Navdeep Jaitly
+,
+Heng Ji
+,
+Alane Suhr
+,
+Yizhe Zhang
+View a PDF of the paper titled Training Software Engineering Agents and Verifiers with SWE-Gym, by Jiayi Pan and 6 other authors
+View PDF
+HTML (experimental)
+Abstract:
+We present SWE-Gym, the first environment for training real-world software engineering (SWE) agents. SWE-Gym contains 2,438 real-world Python task instances, each comprising a codebase with an executable runtime environment, unit tests, and a task specified in natural language. We use SWE-Gym to train language model based SWE agents, achieving up to 19% absolute gains in resolve rate on the popular SWE-Bench Verified and Lite test sets. We also experiment with inference-time scaling through verifiers trained on agent trajectories sampled from SWE-Gym. When combined with our fine-tuned SWE agents, we achieve 32.0% and 26.0% on SWE-Bench Verified and Lite, respectively, reflecting a new state-of-the-art for open-weight SWE agents. To facilitate further research, we publicly release SWE-Gym, models, and agent trajectories.
+Comments:
+Accepted at ICML 2025. Code at
+this https URL
+Subjects:
+Software Engineering (cs.SE)
+; Computation and Language (cs.CL)
+Cite as:
+arXiv:2412.21139
+[cs.SE]
+(or
+arXiv:2412.21139v2
+[cs.SE]
+for this version)
+https://doi.org/10.48550/arXiv.2412.21139
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Jiayi Pan [
+view email
+]
+[v1]
+Mon, 30 Dec 2024 18:15:39 UTC (156 KB)
+[v2]
+Fri, 6 Jun 2025 07:53:20 UTC (295 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Training Software Engineering Agents and Verifiers with SWE-Gym, by Jiayi Pan and 6 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.SE
+< prev
+|
+next >
+new
+|
+recent
+|
+2024-12
+Change to browse by:
+cs
+cs.CL
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/250104519-rstar-math-small-llms-can-master-math-reasoning-with-self-evolved-deep.md

@@ -0,0 +1,196 @@
+---
+title: '[2501.04519] rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved
+  Deep Thinking'
+id: 250104519-rstar-math-small-llms-can-master-math-reasoning-with-self-evolved-deep
+tags:
+- deepread
+created: '2026-06-10T00:40:01.597011Z'
+source: https://arxiv.org/abs/2501.04519
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:40:01.596873Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2501.04519] rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
+Computer Science > Computation and Language
+arXiv:2501.04519
+(cs)
+[Submitted on 8 Jan 2025]
+Title:
+rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
+Authors:
+Xinyu Guan
+,
+Li Lyna Zhang
+,
+Yifei Liu
+,
+Ning Shang
+,
+Youran Sun
+,
+Yi Zhu
+,
+Fan Yang
+,
+Mao Yang
+View a PDF of the paper titled rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking, by Xinyu Guan and 7 other authors
+View PDF
+HTML (experimental)
+Abstract:
+We present rStar-Math to demonstrate that small language models (SLMs) can rival or even surpass the math reasoning capability of OpenAI o1, without distillation from superior models. rStar-Math achieves this by exercising "deep thinking" through Monte Carlo Tree Search (MCTS), where a math policy SLM performs test-time search guided by an SLM-based process reward model. rStar-Math introduces three innovations to tackle the challenges in training the two SLMs: (1) a novel code-augmented CoT data sythesis method, which performs extensive MCTS rollouts to generate step-by-step verified reasoning trajectories used to train the policy SLM; (2) a novel process reward model training method that avoids naïve step-level score annotation, yielding a more effective process preference model (PPM); (3) a self-evolution recipe in which the policy SLM and PPM are built from scratch and iteratively evolved to improve reasoning capabilities. Through 4 rounds of self-evolution with millions of synthesized solutions for 747k math problems, rStar-Math boosts SLMs' math reasoning to state-of-the-art levels. On the MATH benchmark, it improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to 86.4%, surpassing o1-preview by +4.5% and +0.9%. On the USA Math Olympiad (AIME), rStar-Math solves an average of 53.3% (8/15) of problems, ranking among the top 20% the brightest high school math students. Code and data will be available at
+this https URL
+.
+Subjects:
+Computation and Language (cs.CL)
+Cite as:
+arXiv:2501.04519
+[cs.CL]
+(or
+arXiv:2501.04519v1
+[cs.CL]
+for this version)
+https://doi.org/10.48550/arXiv.2501.04519
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Li Lyna Zhang [
+view email
+]
+[v1]
+Wed, 8 Jan 2025 14:12:57 UTC (632 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking, by Xinyu Guan and 7 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.CL
+< prev
+|
+next >
+new
+|
+recent
+|
+2025-01
+Change to browse by:
+cs
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/250104519-sysname-small-llms-can-master-math-reasoning-with-self-evolved-deep-th.md

@@ -0,0 +1,3557 @@
+---
+title: '[2501.04519] \sysname: Small LLMs Can Master Math Reasoning with Self-Evolved
+  Deep Thinking'
+id: 250104519-sysname-small-llms-can-master-math-reasoning-with-self-evolved-deep-th
+tags:
+- deepread
+created: '2026-06-10T00:40:46.873514Z'
+source: https://ar5iv.labs.arxiv.org/html/2501.04519
+source_domain: ar5iv.labs.arxiv.org
+fetched_at: '2026-06-10T00:40:46.873327Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2501.04519] \sysname: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
+\sysname
+: Small LLMs Can Master Math Reasoning
+with Self-Evolved Deep Thinking
+Xinyu Guan
+∗
+Li Lyna Zhang
+∗⋄
+Yifei Liu
+Ning Shang   Youran Sun    Yi Zhu    Fan Yang    Mao Yang
+Microsoft Research Asia
+Abstract
+We present
+\sysname
+to demonstrate that small language models (SLMs) can rival or even surpass the math reasoning capability of OpenAI o1, without distillation from superior models.
+\sysname
+achieves this by exercising “deep thinking” through Monte Carlo Tree Search (MCTS), where a math
+policy SLM
+performs test-time search guided by an SLM-based
+process reward model
+.
+\sysname
+introduces three innovations to tackle the challenges in training the two SLMs:
+(1)
+a novel code-augmented CoT data sythesis method, which performs extensive MCTS rollouts to generate
+step-by-step verified reasoning trajectories
+used to train the policy SLM;
+(2)
+a novel process reward model training method that avoids naïve step-level score annotation, yielding a more effective
+process preference model (PPM)
+;
+(3)
+a
+self-evolution recipe
+in which the policy SLM and PPM are built from scratch and iteratively evolved to improve reasoning capabilities.
+Through 4 rounds of self-evolution with millions of synthesized solutions for 747k math problems,
+\sysname
+boosts SLMs’ math reasoning to state-of-the-art levels. On the MATH benchmark, it improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to 86.4%, surpassing o1-preview by +4.5% and +0.9%. On the USA Math Olympiad (AIME),
+\sysname
+solves an average of 53.3% (8/15) of problems, ranking among the top 20% the brightest high school math students. Code and data will be available at
+https://github.com/microsoft/rStar
+.
+Task
+(pass@1 Acc)
+rStar-Math
+(Qwen-7B)
+rStar-Math
+(Qwen-1.5B)
+rStar-Math
+(Phi3-mini)
+OpenAI
+o1-preview
+OpenAI
+o1-mini
+QWQ
+32B-preview
+GPT-4o
+DeepSeek-V3
+MATH
+90.0
+88.6
+86.4
+85.5
+90.0
+90.6
+76.6
+90.2
+AIME 2024
+53.3
+46.7
+43.3
+44.6
+56.7
+50.0
+9.3
+39.2
+Olympiad Bench
+65.6
+64.6
+60.3
+-
+65.3
+61.2
+43.3
+55.4
+College Math
+60.5
+59.3
+59.1
+-
+57.8
+55.8
+48.5
+58.9
+Omni-Math
+50.5
+48.5
+46.0
+52.5
+60.5
+49.6
+30.5
+35.9
+Table 1:
+\sysname
+enables frontier math reasoning in SLMs via deep thinking over 64 trajectories.
+$*$
+$*$
+footnotetext:
+Equal contribution.
+$\diamond$
+$\diamond$
+footnotetext:
+Project leader; correspondence to lzhani@microsoft.com
+$\S$
+$\S$
+footnotetext:
+Xinyu Guan and Youran Sun did this work during the internship at MSRA. Xinyu Guan (2001gxy@gmail.com) is with Peking University, Youran Sun is with Tsinghua University.
+1
+Introduction
+Recent studies have demonstrated that large language models (LLMs) are capable of tackling mathematical problems
+(Team,
+2024a
+; Yang et al.,
+2024
+; OpenAI,
+2024
+; Liu et al.,
+2024
+)
+. However, the conventional approach of having LLMs generate complete solutions in a single inference – akin to System 1 thinking
+(Daniel,
+2011
+)
+– often yields fast but error-prone results
+(Valmeekam et al.,
+2023
+; OpenAI,
+2023
+)
+. In response, test-time compute scaling
+(Snell et al.,
+2024
+; Qi et al.,
+2024
+)
+suggests a paradigm shift toward a System 2-style thinking, which emulates human reasoning through a slower and deeper thought process. In this paradigm, an LLM serves as a policy model to generate multiple math reasoning steps, which are then evaluated by another LLM acting as a reward model
+(OpenAI,
+2024
+)
+. The steps and solutions deemed more likely to be correct are selected. The process repeats iteratively and ultimately derives the final answer.
+In the test-time compute paradigm, the key is to train a powerful policy model that generates promising solution steps and a reliable reward model that accurately evaluates them, both of which depend on
+high-quality
+training data. Unfortunately, it is well-known that off-the-shelf high-quality math reasoning data is scarce, and synthesizing high-quality math data faces fundamental challenges.
+For the policy model, it is challenging to distinguish erroneous reasoning steps from the correct ones, complicating the elimination of low-quality data. It is worth noting that in math reasoning, a correct final answer does not ensure the correctness of the entire reasoning trace
+(Lanham et al.,
+2023
+)
+. Incorrect intermediate steps significantly decrease data quality.
+As for the reward model, process reward modeling (PRM) shows a great potential by providing fine-grained feedback on intermediate steps
+(Lightman et al.,
+2023
+)
+. However, the training data is even scarcer in this regard: accurate step-by-step feedback requires intense human labeling efforts and is impractical to scale, while those automatic annotation attempts show limited gains due to noisy reward scores
+(Luo et al.,
+2024
+; Wang et al.,
+2024c
+; Chen et al.,
+2024
+)
+.
+Due to the above challenges, existing distill-based data synthesis approaches to training policy models, e.g., scaling up GPT4-distilled CoT data
+(Tang et al.,
+2024
+; Huang et al.,
+2024
+)
+, have shown diminishing returns and cannot exceed the capability of their teacher model; meanwhile, as of today, training reliable PRMs for math reasoning remains an open question.
+Figure 1:
+The overview of
+\sysname
+.
+In this work, we introduce
+\sysname
+, a self-evolvable System 2-style reasoning approach that achieves the state-of-the-art math reasoning, rivaling and sometimes even surpassing OpenAI o1 on challenging math competition benchmarks with a model size as small as 7 billion. Unlike solutions relying on superior LLMs for data synthesis,
+\sysname
+leverages smaller language models (SLMs) with Monte Carlo Tree Search (MCTS) to establish a self-evolutionary process, iteratively generating higher-quality training data. To achieve self-evolution,
+\sysname
+introduces three key innovations.
+First, a novel code-augmented CoT data synthesis method, which performs
+extensive
+MCTS rollouts to generate
+step-by-step verified reasoning trajectories
+with
+self-annotated MCTS Q-values
+. Specifically, math problem-solving is decomposed into multi-step generation within MCTS. At each step, the SLM serving as the policy model samples candidate nodes, each generating a one-step CoT and the corresponding Python code. To verify the generation quality, only nodes with successful Python code execution are retained, thus mitigating errors in intermediate steps. Moreover, extensive MCTS rollouts automatically assign a Q-value to each intermediate step based on its contribution: steps contributing to more trajectories that lead to the correct answer are given higher Q-values and considered higher quality. This ensures that the reasoning trajectories generated by SLMs consist of correct, high-quality intermediate steps.
+Second, a novel method that trains an SLM acting as a
+process preference model
+, i.e., a PPM to implement the desired PRM, that reliably predicts a reward label for each math reasoning step. The PPM leverages the fact that, although Q-values are still not precise enough to score each reasoning step despite using extensive MCTS rollouts, the Q-values can reliably distinguish positive (correct) steps from negative (irrelevant/incorrect) ones. Thus the training method constructs preference pairs for each step based on Q-values and uses a pairwise ranking loss
+(Ouyang et al.,
+2022
+)
+to optimize PPM’s score prediction for each reasoning step, achieving reliable labeling. This approach avoids conventional methods that directly use Q-values as reward labels
+(Luo et al.,
+2024
+; Chen et al.,
+2024
+)
+, which are inherently noisy and imprecise in stepwise reward assignment.
+Finally, a four-round self-evolution recipe that progressively builds both a frontier policy model and PPM from scratch. We begin by curating a dataset of 747k math word problems from publicly available sources. In each round, we use the latest policy model and PPM to perform MCTS, generating increasingly high-quality training data using the above two methods to train a stronger policy model and PPM for next round. Each round achieves progressive refinement: (1) a stronger policy SLM, (2) a more reliable PPM, (3) generating better reasoning trajectories via PPM-augmented MCTS, and (4) improving training data coverage to tackle more challenging and even competition-level math problems.
+Extensive experiments across four SLMs (1.5B-7B) and seven math reasoning tasks demonstrate the effectiveness of
+\sysname
+. Remarkably,
+\sysname
+improves all four SLMs, matching or even surpassing OpenAI o1 on challenging math benchmarks. On MATH benchmark, with 8 search trajectories,
+\sysname
+boosts Qwen2.5-Math-7B from 58.8% to 89.4% and Qwen2.5-Math-1.5B from 51.2% to 87.8%. With 64 trajectories, the scores rise to 90% and 88.4%, outperforming o1-preview by 4.5% and 2.6% and matching o1-mini’s 90%. On the Olympiad-level AIME 2024,
+\sysname
+solves on average 53.3% (8/15) of the problems, exceeding o1-preview by 8.7% and all other open-sourced LLMs. We further conduct comprehensive experiments to verify the superiority of step-by-step verified reasoning trajectories over state-of-the-art data synthesis baselines, as well as the PPM’s effectiveness compared to outcome reward models and Q value-based PRMs. Finally, we present key findings from
+\sysname
+deep thinking, including the intrinsic self-reflection capability and PPM’s preference for theorem-applications intermediate steps.
+2
+Related Works
+Math Data Synthesis
+. Advancements in LLM math reasoning have largely relied on curating high-quality CoT data, with most leading approaches being GPT-distilled, using frontier models like GPT-4 for synthesis
+(Wang et al.,
+2024b
+; Gou et al.,
+2023
+; Luo et al.,
+2023
+)
+. Notable works include NuminaMath
+(Jia LI and Polu,
+2024a
+)
+and
+MetaMath
+(Yu et al.,
+2023b
+)
+. While effective, this limits reasoning to the capabilities of the teacher LLM.
+Hard problems that the teacher LLM cannot solve are excluded in the training set.
+Even solvable problems may contain error-prone intermediate steps, which are hard to detect. Although rejection sampling methods
+(Yuan et al.,
+2023
+; Brown et al.,
+2024
+)
+can improve data quality,
+they do not guarantee correct intermediate steps. As a result, scaling up CoT data has diminishing returns, with gains nearing saturation—e.g., OpenMathInstruct-2
+(Toshniwal et al.,
+2024
+)
+only sees a 3.9% boost on MATH despite an 8× increase in dataset size.
+Scaling Test-time Compute
+has introduced new scaling laws, allowing LLMs to improve performance across by generating multiple samples and using reward models for best-solution selection
+(Snell et al.,
+2024
+; Wu et al.,
+2024
+; Brown et al.,
+2024
+)
+. Various test-time search methods have been proposed
+(Kang et al.,
+2024
+; Wang et al.,
+2024a
+)
+, including random sampling
+(Wang et al.,
+2023
+)
+and tree-search methods
+(Yao et al.,
+2024
+; Hao et al.,
+2023
+; Zhang et al.,
+2024b
+; Qi et al.,
+2024
+)
+like MCTS. However, open-source methods for scaling test-time computation have shown limited gains in math reasoning, often due to policy LLM or reward model limitations.
+\sysname
+addresses this by iteratively evolving the policy LLM and reward model, achieving System 2 mathematical reasoning performance comparable to OpenAI o1
+(OpenAI,
+2024
+)
+.
+Reward Models
+are crucial for effective System 2 reasoning but are challenging to obtain. Recent works include LLM-as-a-Judge for verification
+(Zheng et al.,
+2023
+; Qi et al.,
+2024
+)
+and specialized reward models like Outcome Reward Model
+(Yang et al.,
+2024
+; Yu et al.,
+2023a
+)
+and Process Reward Model (PRM)
+(Lightman et al.,
+2024
+)
+. While PRMs offer promising dense, step-level reward signals for
+complex reasoning
+(Luo et al.,
+2024
+; Wang et al.,
+2024c
+)
+, collecting step-level annotations remains an obstacle. While
+Kang et al. (
+2024
+); Wang et al. (
+2024a
+)
+rely on costly human-annotated datasets like PRM800k
+(Lightman et al.,
+2024
+)
+,
+recent approaches
+(Wang et al.,
+2024c
+; Luo et al.,
+2024
+)
+explore automated annotation via Monte Carlo Sampling or MCTS. However, they struggle to generate precise reward scores, which limits performance gains.
+\sysname
+introduces a novel process preference reward (PPM) that eliminates the need for accurate step-level reward score annotation.
+3
+Methodology
+3.1
+Design Choices
+MCTS for Effective System 2 Reasoning
+.
+We aim to train a math policy SLM and a process reward model (PRM), and integrating both within Monte Carlo Tree Search (MCTS) for System 2 deep thinking. MCTS is chosen for two key reasons. First, it breaks down complex math problems into simpler single-step generation tasks, reducing the difficulty for the policy SLM compared to other System 2 methods like Best-of-N
+(Brown et al.,
+2024
+)
+or self-consistency
+(Wang et al.,
+2023
+)
+, which require generating full solutions in one inference.
+Second, the step-by-step generation in MCTS naturally yields step-level training data for both models. Standard MCTS rollout automatically assign Q-value to each step based on its contribution to the final correct answer, obviating the need for human-generated step-level annotations for process reward model training.
+Ideally, advanced LLMs such as GPT-4 could be integrated within MCTS to generate training data. However, this approach faces two key challenges. First, even these powerful models struggle to consistently solve difficult problems, such as Olympiad-level mathematics. Consequently, the resulting training data would primarily consist of simpler solvable problems, limiting its diversity and quality. Second, annotating per-step Q-values demands extensive MCTS rollouts; insufficient tree exploration can lead to spurious Q-value assignments, such as overestimating suboptimal steps. Given that each rollout involves multiple single-step generations and these models are computationally expensive, increasing rollouts significantly raises inference costs.
+Overview
+. To this end, we explore using two 7B SLMs (a policy SLM and a PRM) to generate higher-quality training data, with their smaller size allowing for extensive MCTS rollouts on accessible hardware (e.g., 4
+×
+\times
+40GB A100 GPUs). However, self-generating data presents greater challenges for SLMs, due to their weaker capabilities.
+SLMs frequently fail to generate correct solutions, and even when the final answer is correct, the intermediate steps are often flawed or of poor quality. Moreover, SLMs solve fewer challenging problems compared to advanced models like GPT-4.
+This section introduces our methodology, as illustrated in Fig.
+1
+. To mitigate errors and low-quality intermediate steps, we introduce a code-augmented CoT synthetic method, which performs extensive MCTS rollouts to generate step-by-step verified reasoning trajectories, annotated with Q-values. To further improve SLM performance on challenging problems, we introduce a four-round self-evolution recipe. In each round, both the policy SLM and the reward model are updated to stronger versions, progressively tackling more difficult problems and generating higher-quality training data. Finally, we present a novel process reward model training approach that eliminates the need for precise per-step reward annotations, yielding the more
+effective process preference model (PPM).
+3.2
+Step-by-Step Verified Reasoning Trajectory
+We start by introducing our method for generating step-by-step verified reasoning trajectories with per-step Q-value annotations. Given a problem
+x
+x
+and a policy model
+M
+M
+, we run the standard MCTS to incrementally construct a search tree for step-by-step solution exploration. As shown in Fig.
+1
+(a),
+the root node represents question
+x
+x
+, while child nodes correspond to intermediate steps
+s
+s
+generated by
+M
+M
+. A root-to-leaf path ending at terminal node
+s
+d
+s_{d}
+forms a trajectory
+𝐭
+=
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+d
+\mathbf{t}=x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{d}
+, with each step
+s
+i
+s_{i}
+assigned a Q-value
+Q
+​
+(
+s
+i
+)
+Q(s_{i})
+.
+From the search tree
+𝒯
+\mathcal{T}
+, we extract solution trajectories
+𝕋
+=
+{
+𝐭
+1
+,
+𝐭
+2
+,
+…
+,
+𝐭
+n
+}
+​
+(
+n
+≥
+1
+)
+\mathbb{T}=\{\mathbf{t}^{1},\mathbf{t}^{2},...,\mathbf{t}^{n}\}(n\geq 1)
+. Our goal is to select high-quality trajectories from
+𝒯
+\mathcal{T}
+to construct the training set. For this purpose, we introduce code-augmented CoT synthesis method to filter out low-quality generations and perform extensive rollouts to improve the reliability of Q-value accuracy.
+Code-augmented CoT Generation
+. Prior MCTS approaches primarily generate natural language (NL) CoTs
+(Qi et al.,
+2024
+; Zhang et al.,
+2024a
+)
+. However, LLMs often suffer from hallucination, producing incorrect or irrelevant steps yet still arrive at the correct answer by chance
+(Lanham et al.,
+2023
+)
+. These flawed steps are challenging to detect and eliminate. To address this, we propose a novel code execution augmented CoT. As shown in Fig.
+2
+, the policy model generates a one-step NL CoT alongside its corresponding Python code, where the NL CoT is embedded as a Python comment. Only generations with successfully executed Python code are retained as valid candidates.
+Figure 2:
+An example of Code-augmented CoT.
+Specifically, starting from the initial root node
+x
+x
+, we perform multiple MCTS iterations through
+selection
+,
+expansion
+,
+rollout
+, and
+back-propagation
+. At step
+i
+i
+, we collect the latest reasoning trajectory
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+i
+−
+1
+x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{i-1}
+as the current state. Based on this state, we prompt (see Appendix
+A.3
+) the policy model to generate
+n
+n
+candidates
+s
+i
+,
+0
+,
+…
+,
+s
+i
+,
+n
+−
+1
+s_{i,0},...,s_{i,n-1}
+for step
+i
+i
+. Python code execution is then employed to filter valid nodes. As shown in Fig.
+2
+, each generation
+s
+i
+,
+j
+s_{i,j}
+is concatenated with the code from all previous steps, forming
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+i
+−
+1
+⊕
+s
+i
+,
+j
+s_{1}\oplus s_{2}\oplus...\oplus s_{i-1}\oplus s_{i,j}
+. Candidates that execute successfully are retained as valid nodes and scored by the PPM, which assigns a Q-value
+q
+​
+(
+s
+i
+)
+q(s_{i})
+.
+Then, we use the well-known Upper Confidence bounds for Trees (UCT)
+(Kocsis and Szepesvári,
+2006
+)
+to select the best node among the
+n
+n
+candidates. This selection process is mathematically represented as:
+UCT
+​
+(
+s
+)
+=
+Q
+​
+(
+s
+)
++
+c
+​
+ln
+⁡
+N
+p
+​
+a
+​
+r
+​
+e
+​
+n
+​
+t
+​
+(
+s
+)
+N
+​
+(
+s
+)
+;
+where
+Q
+​
+(
+s
+)
+=
+q
+​
+(
+s
+)
+N
+​
+(
+s
+)
+\displaystyle\text{UCT}(s)=Q(s)+c\sqrt{\frac{\ln N_{parent}(s)}{N(s)}};\quad\text{where}\quad Q(s)=\frac{q(s)}{N(s)}
+(1)
+where
+N
+​
+(
+s
+)
+N(s)
+denotes the number of visits to node
+s
+s
+, and
+N
+parent
+​
+(
+s
+)
+N_{\text{parent}}(s)
+is the visit count of
+s
+s
+’s parent node. The predicted reward
+q
+​
+(
+s
+)
+q(s)
+is provided by the PPM and will be updated through back-propagation.
+c
+c
+is a constant that balances exploitation and exploration.
+Extensive Rollouts for Q-value Annotation
+. Accurate Q-value
+Q
+​
+(
+s
+)
+Q(s)
+annotation in Eq.
+1
+is crucial for guiding MCTS node selection towards correct problem-solving paths and identifying high-quality steps within trajectories.
+To improve Q-value reliability, we draw inspiration from Go players, who retrospectively evaluate the reward of each move based on game outcomes. Although initial estimates may be imprecise, repeated gameplay refines these evaluations over time. Similarly, in each rollout, we update the Q-value of each step based on its contribution to achieving the correct final answer. After extensive MCTS rollouts, steps consistently leading to correct answers achieve higher Q-values, occasional successes yield moderate Q-values, and consistently incorrect steps receive low Q-values. Specifically, we introduce two self-annotation methods to obtain these step-level Q-values. Fig.
+1
+(c) shows the detailed setting in the four rounds of self-evolution.
+Terminal-guided annotation
+. During the first two rounds, when the PPM is unavailable or insufficiently accurate, we use terminal-guided annotation. Formally, let
+q
+​
+(
+s
+i
+)
+k
+q(s_{i})^{k}
+denote the q value for step
+s
+i
+s_{i}
+after back-propagation in the
+k
+t
+​
+h
+k^{th}
+rollout. Following AlphaGo
+(Silver et al.,
+2017
+)
+and rStar
+(Qi et al.,
+2024
+)
+, we score each intermediate node based on its contribution to the final correct answer:
+q
+​
+(
+s
+i
+)
+k
+=
+q
+​
+(
+s
+i
+)
+k
+−
+1
++
+q
+​
+(
+s
+d
+)
+k
+;
+\displaystyle q(s_{i})^{k}=q(s_{i})^{k-1}+q(s_{d})^{k};
+(2)
+where the initial q value
+q
+​
+(
+s
+i
+)
+0
+=
+0
+q(s_{i})^{0}=0
+in the first rollout. If this step frequently leads to a correct answer, its
+q
+q
+value will increase; otherwise, it decreases. Terminal nodes are scored as
+q
+​
+(
+s
+d
+)
+=
+1
+q(s_{d})=1
+for correct answers and
+q
+​
+(
+s
+d
+)
+=
+−
+1
+q(s_{d})=-1
+otherwise, as shown in Fig.
+1
+.
+PRM-augmented annotation
+. Starting from the third round, we use PPM to score each step for more effective generation. Compared to terminal-guided annotation, which requires multiple rollouts for a meaningful
+q
+q
+value, PPM directly predicts a non-zero initial
+q
+q
+value.
+PPM-augmented MCTS also helps the policy model to generate higher-quality steps, guiding solutions towards correct paths. Formally, for step
+s
+i
+s_{i}
+, PPM predicts an initial
+q
+​
+(
+s
+i
+)
+0
+q(s_{i})^{0}
+value based on the partial trajectory:
+q
+​
+(
+s
+i
+)
+0
+=
+P
+​
+P
+​
+M
+​
+(
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+i
+−
+1
+⊕
+s
+i
+)
+\displaystyle q(s_{i})^{0}=PPM(x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{i-1}\oplus s_{i})
+(3)
+This
+q
+q
+value will be updated based on terminal node’s
+q
+​
+(
+s
+d
+)
+q(s_{d})
+value through MCTS
+back-propagation
+in Eq.
+2
+.
+For terminal node
+s
+d
+s_{d}
+, we do not use PRM for scoring during training data generation. Instead, we assign a more accurate score based on ground truth labels as terminal-guided rewarding.
+3.3
+Process Preference Model
+Process reward models, which provide granular step-level reward signals, is highly desirable for solving challenging math problems. However, obtaining high-quality step-level training data remains an open challenge. Existing methods rely on human annotations
+(Lightman et al.,
+2023
+)
+or MCTS-generated scores
+(Zhang et al.,
+2024a
+; Chen et al.,
+2024
+)
+to assign a score for each step. These scores then serve as training targets, with methods such as MSE loss
+(Chen et al.,
+2024
+)
+or pointwise loss
+(Wang et al.,
+2024c
+; Luo et al.,
+2024
+; Zhang et al.,
+2024a
+)
+used to minimize the difference between predicted and labeled scores.
+As a result, the precision of these annotated step-level reward scores directly determines the effectiveness of the resulting process reward model.
+Unfortunately, precise per-step scoring remains a unsolved challenge. Although our extensive MCTS rollouts improve the reliability of Q-values, precisely evaluating fine-grained step quality presents a major obstacle. For instance, among a set of correct steps, it is difficult to rank them as best, second-best, or average and then assign precise scores. Similarly, among incorrect steps, differentiating the worst from moderately poor steps poses analogous challenges. Even expert human annotation struggles with consistency, particularly at scale, leading to inherent noise in training labels.
+We introduce a novel training method that trains a process preference model (PPM) by constructing step-level positive-negative preference pairs. As shown in Fig.
+1
+(b), instead of using Q-values as direct reward labels, we use them to select steps from MCTS tree for preference pair construction. For each step, we select two candidates with the highest Q-values as positive steps and two with the lowest as negative steps. Critically, the selected positive steps must lead to a correct final answer, while negative steps must lead to incorrect answers. For intermediate steps (except the final answer step), the positive and negative pairs share the same preceding steps. For the final answer step, where identical reasoning trajectories rarely yield different final answers, we relax this restriction.
+We select two correct trajectories with the highest average Q-values as positive examples and two incorrect trajectories with the lowest average Q-values as negative examples. Following
+(Ouyang et al.,
+2022
+)
+, we define our loss function using the standard Bradley-Terry model with a pairwise ranking loss:
+ℒ
+p
+​
+p
+​
+m
+​
+(
+θ
+)
+=
+−
+1
+2
+×
+2
+​
+E
+(
+x
+,
+y
+i
+p
+​
+o
+​
+s
+,
+y
+i
+n
+​
+e
+​
+g
+∈
+𝔻
+)
+​
+[
+l
+​
+o
+​
+g
+​
+(
+σ
+​
+(
+r
+θ
+​
+(
+x
+,
+y
+i
+p
+​
+o
+​
+s
+)
+−
+r
+θ
+​
+(
+x
+,
+y
+i
+n
+​
+e
+​
+g
+)
+)
+)
+]
+\displaystyle\mathcal{L}_{ppm}(\theta)=-\frac{1}{2\times 2}E_{(x,y_{i}^{pos},y_{i}^{neg}\in\mathbb{D})}[log(\sigma(r_{\theta}(x,y_{i}^{pos})-r_{\theta}(x,y_{i}^{neg})))]
+(4)
+when
+i
+is not final answer step
+,
+y
+i
+p
+​
+o
+​
+s
+=
+s
+1
+⊕
+…
+⊕
+s
+i
+−
+1
+⊕
+s
+i
+p
+​
+o
+​
+s
+;
+y
+i
+n
+​
+e
+​
+g
+=
+s
+1
+⊕
+…
+⊕
+s
+i
+−
+1
+⊕
+s
+i
+n
+​
+e
+​
+g
+\displaystyle\text{when $i$ is not final answer step},y_{i}^{pos}=s_{1}\oplus...\oplus s_{i-1}\oplus s_{i}^{pos};y_{i}^{neg}=s_{1}\oplus...\oplus s_{i-1}\oplus s_{i}^{neg}\vskip-4.30554pt
+(5)
+Here,
+r
+θ
+​
+(
+x
+,
+y
+i
+)
+r_{\theta}(x,y_{i})
+denotes the output of the PPM, where
+x
+x
+is the problem and
+y
+y
+is the trajectory from the first step to the
+i
+t
+​
+h
+i^{th}
+step.
+3.4
+Self-Evolved Deep Thinking
+3.4.1
+Training with Step-by-Step Verified Reasoning Trajectory
+Math Problems Collection
+. We collect a large dataset of 747k math word problems with final answer ground-truth labels, primarily from NuminaMath
+(Jia LI and Polu,
+2024a
+)
+and MetaMath
+(Yu et al.,
+2023b
+)
+. Notably, only competition-level problems (e.g., Olympiads and AIME/AMC) from NuminaMath are included, as we observe that grade-school-level problems do not significantly improve LLM complex math reasoning. To augment the limited competition-level problems, we follow
+(Li et al.,
+2024
+)
+and use GPT-4 to synthesize new problems based on the seed problems in 7.5k MATH train set and 3.6k AMC-AIME training split. However, GPT-4 often generated unsolvable problems or incorrect solutions for challenging seed problems. To filter these, we prompt GPT-4 to generate 10 solutions per problem, retaining only those with at least 3 consistent solutions.
+Reasoning Trajectories Collection
+. Instead of using the original solutions in the 747k math dataset, we conduct extensive MCTS rollouts (Sec.
+3.2
+) to generate higher-quality step-by-step verified reasoning trajectories. In each self-evolution round, we perform 16 rollouts per math problem, which leads to 16 reasoning trajectories. Problems are then categories by difficulty based on the correct ratio of the generated trajectories:
+easy
+(all solutions are correct),
+medium
+(a mix of correct and incorrect solutions) and
+hard
+(all solutions are incorrect). For
+hard
+problems with no correct trajectories, an additional MCTS with 16 rollouts is performed. After that, all step-by-step trajectories and their annotated Q-values are collected and filtered to train the policy SLM and process preference model.
+Supervised Fine-tuning the Policy SLM
+. Through extensive experiments, we find that selecting high-quality reasoning trajectories is the key for fine-tuning a frontier math LLM. While methods such as GPT-distillation and Best-of-N can include low-quality or erroneous intermediate steps, a more effective approach ensures that every step in the trajectory is of high quality. To achieve this, we use per-step Q-values to select optimal trajectories from MCTS rollouts. Specifically, for each math problem, we select the top-2 trajectories with the highest average Q-values among those leading to correct answers as SFT training data.
+Training PPM
+. The PPM is initialized from the fine-tuned policy model, with its next-token prediction head replaced by a scalar-value head consisting of a linear layer and a tanh function to constrain outputs to the range [-1, 1]. We filter out math problems where all solution trajectories are fully correct or incorrect. For problems with mixed outcomes, we select two positive and two negative examples for each step based on Q-values, which are used as preference pairs for training data.
+3.4.2
+Recipe for Self-Evolution
+Table 2:
+Percentage of the 747k math problems correctly solved in each round. Only problems have correct solutions are included in the training set. The first round uses DeepSeek-Coder-Instruct as the policy LLM, while later rounds use our fine-tuned 7B policy SLM.
+#
+models in MCTS
+GSM-level
+MATH-level
+Olympiad-level
+All
+Round 1
+DeepSeek-Coder-V2-Instruct
+96.61%
+67.36%
+20.99%
+60.17%
+Round 2
+policy SLM-r1
+97.88%
+67.40%
+56.04%
+66.60%
+Round 3
+policy SLM-r2, PPM-r2
+98.15%
+88.69%
+62.16%
+77.86%
+Round 4
+policy SLM-r3, PPM-r3
+98.15%
+94.53%
+80.58%
+90.25%
+Table 3:
+Pass@1 accuracy of the resulting policy SLM in each round, showing continuous improvement until surpassing the bootstrap model.
+Round#
+MATH
+AIME 2024
+AMC 2023
+Olympiad Bench
+College Math
+GSM8K
+GaokaoEn 2023
+DeepSeek-Coder-V2-Instruct
+(bootstrap model)
+75.3
+13.3
+57.5
+37.6
+46.2
+94.9
+64.7
+Base (Qwen2.5-Math-7B)
+58.8
+0.0
+22.5
+21.8
+41.6
+91.6
+51.7
+\hdashline
+policy SLM-r1
+69.6
+3.3
+30.0
+34.7
+44.5
+88.4
+57.4
+policy SLM-r2
+73.6
+10.0
+35.0
+39.0
+45.7
+89.1
+59.7
+policy SLM-r3
+75.8
+16.7
+45.0
+44.1
+49.6
+89.3
+62.8
+policy SLM-r4
+78.4
+26.7
+47.5
+47.1
+52.5
+89.7
+65.7
+Table 4:
+The quality of PPM consistently improves across rounds. The policy model has been fixed with policy SLM-r1 for a fair comparison.
+Round#
+MATH
+AIME 2024
+AMC 2023
+Olympiad Bench
+College Math
+GSM8K
+GaokaoEn 2023
+PPM-r1
+75.2
+10.0
+57.5
+35.7
+45.4
+90.9
+60.3
+PPM-r2
+84.1
+26.7
+75.0
+52.7
+54.2
+93.3
+73.0
+PPM-r3
+85.2
+33.3
+77.5
+59.5
+55.6
+93.9
+76.6
+PPM-r4
+87.0
+43.3
+77.5
+61.5
+56.8
+94.2
+77.8
+Due to the weaker capabilities of SLMs, we perform four rounds of MCTS deep thinking to progressively generate higher-quality data and expand the training set with more challenging math problems. Each round uses MCTS to generate step-by-step verified reasoning trajectories, which are then used to train the new policy SLM and PPM. The new models are then applied in next round to generate higher-quality training data. Fig.
+1
+(c) and Table
+2
+detail the models used for data generation in each round, along with the identifiers of the trained policy model and PPM. Next, we outline the details and specific improvements targeted in each round.
+Round 1: Bootstrapping an initial strong policy SLM-r1
+. To enable SLMs to self-generate reasonably good training data, we perform a bootstrap round to fine-tune an initial strong policy model, denoted as SLM-r1.
+As shown in Table
+2
+, we run MCTS with DeepSeek-Coder-V2-Instruct (236B) to collect the SFT data. With no available reward model in this round, we use terminal-guided annotation for Q-values and limit MCTS to 8 rollouts for efficiency. For correct solutions, the top-2 trajectories with the highest average Q-values are selected as SFT data. We also train PPM-r1, but the limited rollouts yields unreliable Q-values, affecting the effectiveness of PPM-r1 ( Table
+4
+).
+Round 2: Training a reliable PPM-r2
+. In this round, with the policy model updated to the 7B SLM-r1, we conduct extensive MCTS rollouts for more reliable Q-value annotation and train the first reliable reward model, PPM-r2. Specifically, we perform 16 MCTS rollouts per problem. The resulting step-by-step verified reasoning trajectories show significant improvements in both quality and Q-value precision. As shown in Table
+4
+, PPM-r2 is notably more effective than in the bootstrap round. Moreover, the policy SLM-r2 also continues to improve as expected (Table
+3
+).
+Round 3: PPM-augmented MCTS to significantly improve data quality
+. With the reliable PPM-r2, we perform PPM-augmented MCTS in this round to generate data, leading to significantly higher-quality trajectories that cover more math and Olympiad-level problems in the training set (Table
+2
+). The generated reasoning trajectories and self-annotated Q-values are then used to train the new policy SLM-r3 and PPM-r3, both of which show significant improvements.
+Round 4: Solving challenging math problems
+. After the third round, while grade school and MATH problems achieve high success rates, only 62.16% of Olympiad-level problems are included in the training set. This is
+NOT
+solely due to weak reasoning abilities in our SLMs, as many Olympiad problems remain unsolved by GPT-4 or o1. To improve coverage, we adopt a straightforward strategy. For unsolved problems after 16 MCTS rollouts, we perform an additional 64 rollouts, and if needed, increase to 128. We also conduct multiple MCTS tree expansions with different random seeds. This boosts the success rate of Olympiad-level problems to 80.58%.
+After four rounds of self-evolution, 90.25% of the 747k math problems are successfully covered into the training set, as shown in Table
+2
+. Among the remaining unsolved problems, a significant portion consists of synthetic questions. We manually review a random sample of 20 problems and find that 19 are incorrectly labeled with wrong answers. Based on this, we conclude that the remaining unsolved problems are of low quality and thus terminate the self-evolution at round 4.
+4
+Evaluation
+4.1
+Setup
+Evaluation Datasets
+. We evaluate
+\sysname
+on diverse mathematical benchmarks. In addition to the widely-used GSM8K
+(Cobbe et al.,
+2021
+)
+, we include challenging benchmarks from multiple domains:
+(i)
+competition and Olympiad-level benchmarks, such as MATH-500
+(Lightman et al.,
+2023
+)
+, AIME 2024
+(AI-MO,
+2024a
+)
+, AMC 2023
+(AI-MO,
+2024b
+)
+and Olympiad Bench
+(He et al.,
+2024
+)
+. Specifically, AIME is the exams designed to challenge the brightest high school math students in American, with the 2024 dataset comprising 30 problems from AIME I and II exams;
+(ii)
+college-level math problems from College Math
+(Tang et al.,
+2024
+)
+and
+(iii)
+out-of-domain math benchmark: GaoKao (Chinese
+College Entrance Exam) En 2023
+(Liao et al.,
+2024
+)
+.
+Base Models and Setup
+.
+\sysname
+is a general approach applicable to various LLMs. To show its effectiveness and generalizability, we use SLMs of different sizes as the base policy models:
+Qwen2.5-Math-1.5B
+(Qwen,
+2024b
+)
+, Phi3-mini-Instruct (3B)
+(Microsoft,
+2024
+; Abdin et al.,
+2024
+)
+, Qwen2-Math-7B
+(Qwen,
+2024a
+)
+and Qwen2.5-Math-7B
+(Qwen,
+2024c
+)
+. Among these, Phi3-mini-Instruct is a general-purpose SLM without specialization in math reasoning.
+Due to limited GPU resources, we performed 4 rounds of self-evolution exclusively on Qwen2.5-Math-7B, yielding 4 evolved policy SLMs (Table
+3
+) and 4 PPMs (Table
+4
+). For the other 3 policy LLMs, we fine-tune them using step-by-step verified trajectories generated from Qwen2.5-Math-7B’s 4th round. The final PPM from this round is then used as the reward model for the 3 policy SLMs.
+Baselines
+.
+\sysname
+is a System 2 method. We compare it against three strong baselines representing both System 1 and System 2 approaches:
+(i)
+Frontier LLMs
+, including GPT-4o, the latest Claude, OpenAI o1-preview and o1-mini.
+We measure their accuracy on AMC 2023, Olympiad Bench, College Math, Gaokao and GSM8K, with accuracy numbers for other benchmarks are taken from public technical reports
+(Team,
+2024a
+)
+.
+(ii)
+Open-sourced superior reasoning models
+, including DeepSeek-Coder-v2-Instruct, Mathstral
+(Team,
+2024b
+)
+, NuminaMath-72B
+(Jia LI and Polu,
+2024a
+)
+, and LLaMA3.1
+(Dubey et al.,
+2024
+)
+, which represent the current mainstream System 1 approaches for improving LLM math reasoning.
+(iii)
+Both System 1 and System 2 performance of the base models trained from the original models teams
+, including Instruct versions (e.g., Qwen2.5-Math-7B-Instruct) and Best-of-N (e.g., Qwen2.5-Math-72B-Instruct+Qwen2.5-Math-RM-72B). Notably, the reward model used for the three Qwen base models is a 72B ORM, significantly larger than our 7B PPM.
+Evaluation Metric
+. We report Pass@1 accuracy for all baselines. For System 2 baselines, we use default evaluation settings, such as default thinking time for o1-mini and o1-preview. For Qwen models with Best-of-N, we re-evaluate MATH-500, AIME/AMC accuracy; other benchmarks results are from their technical reports. For a fair comparison,
+\sysname
+run MCTS to generate the same number of solutions as Qwen. Specifically, for AIME/AMC, we generate 16 trajectories for AIME/AMC and 8 for other benchmarks, using PPM to select the best solution. We also report performance with increased test-time computation using 64 trajectories, denoted as
+\sysname
+64
+.
+Table 5:
+The results of
+\sysname
+and other frontier LLMs on the most challenging math benchmarks.
+\sysname
+64
+shows the Pass@1 accuracy achieved when sampling 64 trajectories.
+Competition and College Level
+OOD
+Model
+Method
+MATH
+AIME
+2024
+AMC
+2023
+Olympiad
+Bench
+College
+Math
+GSM8K
+Gaokao
+En 2023
+Frontier LLMs
+GPT-4o
+System 1
+76.6
+9.3
+47.5
+43.3
+48.5
+92.9
+67.5
+Claude3.5-Sonnet
+System 1
+78.3
+16.0
+-
+-
+-
+96.4
+-
+GPT-o1-preview
+-
+85.5
+44.6
+90.0
+-
+-
+-
+-
+GPT-o1-mini
+-
+90.0
+56.7
+95.0
+65.3
+57.8
+94.8
+78.4
+Open-Sourced Reasoning LLMs
+DeepSeek-Coder-V2-Instruct
+System 1
+75.3
+13.3
+57.5
+37.6
+46.2
+94.9
+64.7
+Mathstral-7B-v0.1
+System 1
+57.8
+0.0
+37.5
+21.5
+33.7
+84.9
+46.0
+NuminaMath-72B-CoT
+System 1
+64.0
+3.3
+70.0
+32.6
+39.7
+90.8
+58.4
+LLaMA3.1-8B-Instruct
+System 1
+51.4
+6.7
+25.0
+15.4
+33.8
+76.6
+38.4
+LLaMA3.1-70B-Instruct
+System 1
+65.4
+23.3
+50.0
+27.7
+42.5
+94.1
+54.0
+Qwen2.5-Math-72B-Instruct
+System 1
+85.6
+30.0
+70.0
+49.0
+49.5
+95.9
+71.9
+Qwen2.5-Math-72B-Instruct+72B ORM
+System 2
+85.8
+36.7
+72.5
+54.5
+50.6
+96.4
+76.9
+General Base Model: Phi3-mini-Instruct (3.8B)
+Phi3-mini-Instruct (base model)
+System 1
+41.4
+3.33
+7.5
+12.3
+33.1
+85.7
+37.1
+\sysname
+(3.8B SLM+7B PPM)
+System 2
+85.4
+40.0
+77.5
+59.3
+58.0
+94.5
+77.1
+\sysname
+64
+(3.8B SLM+7B PPM)
+System 2
+86.4
+43.3
+80.0
+60.3
+59.1
+94.7
+77.7
+Math-Specialized Base Model: Qwen2.5-Math-1.5B
+Qwen2.5-Math-1.5B (base model)
+System 1
+51.2
+0.0
+22.5
+16.7
+38.4
+74.6
+46.5
+Qwen2.5-Math-1.5B-Instruct
+System 1
+60.0
+10.0
+60.0
+38.1
+47.7
+84.8
+65.5
+Qwen2.5-Math-1.5B-Instruct+72B ORM
+System 2
+83.4
+20.0
+72.5
+47.3
+50.2
+94.1
+73.0
+\sysname
+(1.5B SLM+7B PPM)
+System 2
+87.8
+46.7
+80.0
+63.5
+59.0
+94.3
+77.7
+\sysname
+64
+(1.5B SLM+7B PPM)
+System 2
+88.6
+46.7
+85.0
+64.6
+59.3
+94.8
+79.5
+Math-Specialized Base Model: Qwen2-Math-7B
+Qwen2-Math-7B (base model)
+System 1
+53.4
+3.3
+25.0
+17.3
+39.4
+80.4
+47.3
+Qwen2-Math-7B-Instruct
+System 1
+73.2
+13.3
+62.5
+38.2
+45.9
+89.9
+62.1
+Qwen2-Math-7B-Instruct+72B ORM
+System 2
+83.4
+23.3
+62.5
+47.6
+47.9
+95.1
+71.9
+\sysname
+(7B SLM+7B PPM)
+System 2
+88.2
+43.3
+80.0
+63.1
+58.4
+94.6
+78.2
+\sysname
+64
+(7B SLM+7B PPM)
+System 2
+88.6
+46.7
+85.0
+63.4
+59.3
+94.8
+79.2
+Math-Specialized Base Model: Qwen2.5-Math-7B
+Qwen2.5-Math-7B (base model)
+System 1
+58.8
+0.0
+22.5
+21.8
+41.6
+91.6
+51.7
+Qwen2.5-Math-7B-Instruct
+System 1
+82.6
+6.0
+62.5
+41.6
+46.8
+95.2
+66.8
+Qwen2.5-Math-7B-Instruct+72B ORM
+System 2
+88.4
+26.7
+75.0
+49.9
+49.6
+97.9
+75.1
+\sysname
+(7B SLM+7B PPM)
+System 2
+89.4
+50.0
+87.5
+65.3
+59.0
+95.0
+80.5
+\sysname
+64
+(7B SLM+7B PPM)
+System 2
+90.0
+53.3
+87.5
+65.6
+60.5
+95.2
+81.3
+4.2
+Main Results
+Results on diverse challenging math benchmarks
+. Table
+5
+shows the results of
+\sysname
+with comparing to state-of-the-art reasoning models. We highlight three key observations:
+(1)
+\sysname
+significantly improves SLMs math reasoning capabilities, achieving performance comparable to or surpassing OpenAI o1 with substantially smaller model size (1.5B-7B). For example, Qwen2.5-Math-7B, originally at 58.8% accuracy on MATH, improved dramatically to 90.0% with
+\sysname
+, outperforming o1-preview and Claude 3.5 Sonnet while matching o1-mini. On the College Math benchmark,
+\sysname
+exceeds o1-mini by 2.7%. On AIME 2024,
+\sysname
+scored 53.3%, ranking just below o1-mini, with the 7B model solving 8/15 problems in both AIME I and II, placing in the top 20% of the brightest high school math students.
+Notably, 8 of the unsolved problems were geometry-based, requiring visual understanding, a capability
+\sysname
+currently does not support.
+(2)
+Despite using smaller policy models (1.5B-7B) and reward models (7B),
+\sysname
+significantly outperforms state-of-the-art System 2 baselines. Compared to Qwen Best-of-N baselines, which use the same base models (Qwen2-Math-7B, Qwen2.5-Math-1.5B/7B) but a 10
+×
+\times
+larger reward model (Qwen2.5-Math-RM-72B),
+\sysname
+consistently improves the reasoning accuracy of all base models to state-of-the-art levels. Even against Best-of-N with a 10
+×
+\times
+larger Qwen2.5-Math-72B-Instruct policy model,
+\sysname
+surpasses it on all benchmarks except GSM8K, using the same number of sampled solutions.
+(3)
+Beyond well-known benchmarks like MATH, GSM8K, and AIME, which may risk over-optimization,
+\sysname
+shows strong generalizability on other challenging math benchmarks, including Olympiad Bench, College Math, and the Chinese College Entrance Math Exam (Gaokao), setting new state-of-the-art scores. As discussed in Sec.
+3.4
+, our training set is primarily sourced from public datasets, with no specific optimizations for these benchmarks.
+Figure 3:
+Reasoning performance under scaling up the test-time compute.
+Scaling up test-time computation
+.
+\sysname
+uses MCTS to augment the policy model, searching solutions guided by the PPM. By increasing test-time computation, it explores more trajectories, potentially improving performance.
+In Fig.
+3
+, we show the impact of test-time compute scaling by comparing the accuracy of the official Qwen Best-of-N across different numbers of sampled trajectories on four challenging math benchmarks. Sampling only one trajectory corresponds to the policy LLM’s Pass@1 accuracy, indicating a fallback to System 1 reasoning. We highlight two key observations:
+(1)
+With only 4 trajectories,
+\sysname
+significantly outperforms Best-of-N baselines, exceeding o1-preview and approaching o1-mini, demonstrating its effectiveness.
+(2)
+Scaling test-time compute improves reasoning accuracy across all benchmarks, though with varying trends. On Math, AIME, and Olympiad Bench,
+\sysname
+shows saturation or slow improvement at 64 trajectories, while on College Math, performance continues to improve steadily.
+4.3
+Ablation Study and Analysis
+We ablate the effectiveness of our three innovations. For System 2-style inference, Pass@1 accuracy is measured with 16 trajectories for AIME and AMC, and 8 for other benchmarks.
+Table 6:
+The continuously improved math reasoning capabilities through
+\sysname
+self-evolved deep thinking. Starting from round 2, the 7B base model powered by
+\sysname
+surpasses GPT-4o.
+Round#
+MATH
+AIME 2024
+AMC 2023
+Olympiad Bench
+College Math
+GSM8K
+GaokaoEn 2023
+GPT-4o
+76.6
+9.3
+47.5
+43.3
+48.5
+92.9
+67.5
+Base 7B model
+58.8
+0.0
+22.5
+21.8
+41.6
+91.6
+51.7
+\sysname
+Round 1
+75.2
+10.0
+57.5
+35.7
+45.4
+90.9
+60.3
+\sysname
+Round 2
+86.6
+43.3
+75.0
+59.4
+55.6
+94.0
+76.4
+\sysname
+Round 3
+87.0
+46.7
+80.0
+61.6
+56.5
+94.2
+77.1
+\sysname
+Round 4
+89.4
+50.0
+87.5
+65.3
+59.0
+95.0
+80.5
+The effectiveness of self-evolution
+. The impressive results in Table
+5
+are achieved after 4 rounds of
+\sysname
+self-evolved deep thinking. Table
+6
+shows the math reasoning performance in each round, demonstrating a continuous improvement in accuracy.
+In round 1, the main improvement comes from applying SFT to the base model. Round 2 brings a significant boost with the application of a stronger PPM in MCTS, which unlocks the full potential of System 2 deep reasoning. Notably, starting from round 2,
+\sysname
+outperforms GPT-4o. Rounds 3 and 4 show further improvements, driven by stronger System 2 reasoning through better policy SLMs and PPMs.
+The effectiveness of step-by-step verified reasoning trajectory
+.
+\sysname
+generates step-by-step verified reasoning trajectories, which eliminate error intermediate steps and further expand training set with more challenging problems. To evaluate its effectiveness, we use the data generated from round 4 as SFT training data and compare it against
+three strong baselines:
+(i)
+GPT-distillation, which includes open-sourced CoT solutions synthesized using GPT-4, such as MetaMath
+(Yu et al.,
+2023b
+)
+, NuminaMath-CoT
+(Jia LI and Polu,
+2024b
+)
+;
+(ii)
+Random sampling from self-generation,
+which use the same policy model (i.e., policy SLM-r3) to randomly generate trajectories;
+(iii)
+Rejection sampling, where 32 trajectories are randomly sampled from the policy model, with high-quality solutions ranked by our trained ORM (appendix
+A.1
+). For fairness, we select two correct trajectories for each math problem in baseline (ii) and (iii). All SFT experiments use the same training recipe.
+Table 7:
+Ablation study on the effectiveness of our step-by-step verified reasoning trajectories as the SFT dataset. We report the SFT accuracy of Qwen2.5-Math-7B fine-tuned with different datasets.
+Dataset
+MATH
+AIME
+AMC
+Olympiad Bench
+College Math
+GSM8K
+GaokaoEn 2023
+GPT-4o
+-
+76.6
+9.3
+47.5
+43.3
+48.5
+92.9
+67.5
+GPT4-distillation
+(Open-sourced)
+MetaMath
+55.2
+3.33
+32.5
+19.1
+39.2
+85.1
+43.6
+NuminaMath-CoT
+69.6
+10.0
+50.0
+37.2
+43.4
+89.8
+59.5
+Self-generation
+by policy SLM-r3
+Random sample
+72.4
+10.0
+45.0
+41.0
+48.0
+87.5
+57.1
+Rejection sampling
+73.4
+13.3
+47.5
+44.7
+50.8
+89.3
+61.7
+Step-by-step verified (ours)
+78.4
+26.7
+47.5
+47.1
+52.5
+89.7
+65.7
+Table
+7
+shows the math reasoning accuracy of Qwen2.5-Math-7B fine-tuned on different datasets. We highlight two observations:
+(i)
+Fine-tuning with our step-by-step verified trajectories significantly outperforms all other baselines. This is primarily due to our PPM-augmented MCTS for code-augmented CoT synthesis, which provides denser verification during math solution generation. It proves more effective than both random sampling, which lacks verification, and rejection sampling, where ORM provides only sparse verification.
+(ii)
+Even randomly sampled code-augmented CoT solutions from our SLM yields comparable or better performance than GPT-4 synthesized NuminaMath and MetaMath datasets.
+This indicates that our policy SLMs, after rounds of self-evolution, can generate high-quality math solutions. These results demonstrates the huge potential of our method to self-generate higher-quality reasoning data without relying on advanced LLM distillation.
+The effectiveness of PPM
+. We train both a strong ORM and Q-value score-based PRM (PQM) for comparison. To ensure a fair evaluation, we use the highest-quality training data: the step-by-step verified trajectories generated in round 4, with selected math problems matching those used for PPM training. Similar to PPM, we use step-level Q-values as to select positive and negative trajectories for each math problem.
+The ORM is trained using a pairwise ranking loss
+(Ouyang et al.,
+2022
+)
+, while the PQM follows
+(Chen et al.,
+2024
+; Zhang et al.,
+2024a
+)
+to use Q-values as reward labels and optimize with MSE loss. Detailed training settings are provided in Appendix
+A.1
+.
+Table 8:
+Ablation study on the reward model. Process reward models (PQM and PPM) outperform ORM, with PPM pushing the frontier of math reasoning capabilities.
+RM
+Inference
+MATH
+AIME
+AMC
+Olympiad Bench
+College Math
+GSM8K
+GaokaoEn
+o1-mini
+-
+90.0
+56.7
+95.0
+65.3
+55.6
+94.8
+78.6
+ORM
+Best-of-N
+82.6
+26.7
+65.0
+55.1
+55.5
+92.3
+72.5
+PQM
+MCTS
+88.2
+46.7
+85.0
+62.9
+57.6
+94.6
+79.5
+PPM
+MCTS
+89.4
+50.0
+87.5
+65.3
+59.0
+95.0
+80.5
+Table
+8
+compares the performance of ORM, PQM, and PPM for System 2 reasoning using our final round policy model. ORM provides reward signals only at the end of problem solving, so we use the Best-of-N method, while PRM and PPM leverage MCTS-driven search. As shown in Table
+8
+, both PQM and PPM outperform ORM by providing denser step-level reward signals, leading to higher accuracy on complex math reasoning tasks. However, PQM struggles on more challenging benchmarks, such as MATH and Olympiad Bench, due to the inherent imprecision of Q-values.
+In contrast, PPM constructs step-level preference data for training, enabling our 7B policy model to achieve comparable or superior performance to o1-mini across all benchmarks.
+5
+Findings and Discussions
+Figure 4:
+An example of intrinsic self-reflection during
+\sysname
+deep thinking.
+The emergence of intrinsic self-reflection capability
+. A key breakthrough in OpenAI o1 is its intrinsic self-reflection capability. When the model makes an error, it recognizes the mistake and can self-correct with a correct answer
+(Noam Brown and Lightman,
+2024
+)
+. Yet it has consistently
+been found to be largely ineffective in open-sourced LLMs. The community has actively explored various approaches, including self-correction
+(Huang et al.,
+2023
+; Kumar et al.,
+2024
+)
+, self-reflection
+(Renze and Guven,
+2024
+; Shinn et al.,
+2024
+)
+, to explicitly train or prompt LLMs to develop such capability.
+In our experiments, we unexpectedly observe that our MCTS-driven deep thinking exhibits self-reflection during problem-solving. As shown in Fig.
+4
+, the model initially formalizes an equation using
+SymPy
+in the first three steps, which would lead to an incorrect answer (left branch). Interestingly, in the fourth step (right branch), the policy model recognizes the low quality of its earlier steps and refrains from continuing along the initial problem-solving path. Instead, it backtracks and resolves the problem using a new, simpler approach, ultimately arriving at the correct answer. An additional example of self-correction is provided in Appendix
+A.2
+. Notably, no self-reflection training data or prompt was included, suggesting that advanced System 2 reasoning can foster intrinsic self-reflection.
+Figure 5:
+Pass@1 accuracy of policy models and their accuracy after applying System 2 reasoning with various reward models, shows that reward models primarily determine the final performance.
+PPM shapes the reasoning boundary in System 2 deep thinking
+. Both the policy and reward models are crucial for System 2 deep reasoning. Our experiments show that once the policy model attains a reasonably strong capability level,
+(see Appendix
+A.1
+), the PPM becomes the key determinant of the upper performance limit.
+Fig.
+5
+summarizes the accuracy of policy models of different sizes, as well as the improvements achieved with reward models. Despite variations in Pass@1 accuracy due to differences in training strategies, datasets, and model scales, the reward model proves to be the dominant factor in System 2 reasoning. For instance, although the SFT accuracy of
+\sysname
+-7B is lower than Qwen2.5-Math-72B-Instruct, pairing it with our 7B PPM allows
+\sysname
+to outperform the 72B policy model with Qwen 72B ORM. Moreover, despite varying Pass@1 accuracy across our three policy SLM sizes, the final reasoning accuracy converges after applying the PPM.
+PPM spots theorem-application steps
+. When solving challenging math problems, identifying and applying relevant theorems or key conclusions often form the cornerstone of successful problem-solving
+(Xin et al.,
+2024
+)
+. In our experiments, we find that during
+\sysname
+problem-solving, our PPM effectively identifies critical theorem-application intermediate steps within policy model’s deep thinking process. These steps are predicted with high reward scores, guiding the policy model to generate the correct solution. Appendix
+A.2
+provides examples where the PPM successfully identifies key theorems such as Fermat’s little theorem
+(Weisstein,
+a
+)
+, Vieta’s formulas
+(Weisstein,
+b
+)
+, the AM-GM inequality
+(
+amg,
+)
+, the Pythagorean theorem
+(
+pyt,
+)
+, and the Shoelace Theorem
+(
+sho,
+)
+, etc.
+Generalization discussions
+.
+\sysname
+offers a general methodology for improving LLM reasoning applicable to various domains. First,
+\sysname
+can generalize to more challenging math tasks, such as theorem proving, though its current focus is on word problems due to dataset limitations. Nonetheless,
+\sysname
+demonstrates the potential to prove mathematical statements. As shown in Appendix
+A.2
+, it successfully proves an Olympiad-level problem involving Fermat’s Little Theorem, providing a step-by-step correct proof through its deep reasoning process. Second,
+\sysname
+can generalize to other domains, such as code and commonsense reasoning. Notably, synthesizing step-by-step verified training trajectories for general reasoning requires a mechanism to provide feedback on whether a given trajectory reaches the desired output at the end of MCTS rollout. For instance, in code reasoning, this could involve designing extensive test cases; in general reasoning, feedback could be obtained through human labeling or mutual verification with another LLM
+(Qi et al.,
+2024
+)
+.
+6
+Conclusion
+In this work, we present
+\sysname
+, a self-evolved System 2 deep thinking approach that significantly boosts the math reasoning capabilities of small LLMs, achieving state-of-the-art OpenAI o1-level performance. Our approach demonstrates that SLMs can self-generate high-quality training data for frontier-level math reasoning. Extensive experiments across four different-sized SLMs and challenging math benchmarks demonstrate the superiority of
+\sysname
+, with achieving leading results while outperforming existing math reasoning LLMs and Best-of-N baselines. We also reveal key findings, including the emergence of self-reflection and the effectiveness of the PPM in identifying critical intermediate steps, such as theorem-application steps. Finally,
+\sysname
+can achieve further improvements by collecting more challenging math problems, we leave this as future work.
+Acknowledgement
+In the early stages of this work, we faced significant challenges due to limited GPU resources and restricted access to the GPT-4 API. We are deeply grateful to Qiufeng Yin and Chengmin Chi for their assistance in collecting math problems and providing GPT-4 resources for new math problem synthesis. Special thanks go to my colleagues, Lingxiao Ma, Ying Cao, Baotong Lu, Jing Liu, Jiahang Xu, Chengruidong Zhang, Siyuan Wang, Gaokai Zhang, Yujian Li, and Yang Wang, for generously sharing their GPU quotas with us.
+References
+[1]
+Inequality of arithmetic and geometric means.
+URL
+https://artofproblemsolving.com/wiki/index.php/AM-GM_Inequality
+.
+[2]
+Pythagorean theorem.
+URL
+https://en.wikipedia.org/wiki/Pythagorean_theorem
+.
+[3]
+Shoelace theorem.
+URL
+https://artofproblemsolving.com/wiki/index.php/Shoelace_Theorem
+.
+Abdin et al. [2024]
+Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah,
+Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl,
+et al.
+Phi-3 technical report: A highly capable language model locally on
+your phone.
+arXiv preprint arXiv:2404.14219
+, 2024.
+AI-MO [2024a]
+AI-MO.
+Aime 2024, 2024a.
+URL
+https://huggingface.co/datasets/AI-MO/aimo-validation-aime
+.
+AI-MO [2024b]
+AI-MO.
+Amc 2023, 2024b.
+URL
+https://huggingface.co/datasets/AI-MO/aimo-validation-amc
+.
+Brown et al. [2024]
+Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le,
+Christopher Ré, and Azalia Mirhoseini.
+Large language monkeys: Scaling inference compute with repeated
+sampling.
+arXiv preprint arXiv:2407.21787
+, 2024.
+Chen et al. [2024]
+Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan.
+Alphamath almost zero: process supervision without process, 2024.
+Cobbe et al. [2021]
+Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz
+Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano,
+et al.
+Training verifiers to solve math word problems.
+arXiv preprint arXiv:2110.14168
+, 2021.
+Daniel [2011]
+Kahneman Daniel.
+Thinking, fast and slow.
+Macmillan
+, 2011.
+Dubey et al. [2024]
+Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad
+Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan,
+et al.
+The llama 3 herd of models.
+arXiv preprint arXiv:2407.21783
+, 2024.
+Gou et al. [2023]
+Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan,
+Weizhu Chen, et al.
+Tora: A tool-integrated reasoning agent for mathematical problem
+solving.
+arXiv preprint arXiv:2309.17452
+, 2023.
+Hao et al. [2023]
+Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and
+Zhiting Hu.
+Reasoning with language model is planning with world model.
+arXiv preprint arXiv:2305.14992
+, 2023.
+He et al. [2024]
+Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen,
+Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al.
+Olympiadbench: A challenging benchmark for promoting agi with
+olympiad-level bilingual multimodal scientific problems.
+arXiv preprint arXiv:2402.14008
+, 2024.
+Huang et al. [2023]
+Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu,
+Xinying Song, and Denny Zhou.
+Large language models cannot self-correct reasoning yet.
+arXiv preprint arXiv:2310.01798
+, 2023.
+Huang et al. [2024]
+Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern,
+Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu.
+O1 replication journey – part 2: Surpassing o1-preview through
+simple distillation big progress or bitter lesson?
+Github
+, 2024.
+URL
+https://github.com/GAIR-NLP/O1-Journey
+.
+Jia LI and Polu [2024a]
+Lewis Tunstall Ben Lipkin Roman Soletskyi Shengyi Costa Huang Kashif Rasul
+Longhui Yu Albert Jiang Ziju Shen Zihan Qin Bin Dong Li Zhou Yann Fleureau
+Guillaume Lample Jia LI, Edward Beeching and Stanislas Polu.
+Numinamath.
+[https://github.com/project-numina/aimo-progress-prize](https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf)
+,
+2024a.
+Jia LI and Polu [2024b]
+Lewis Tunstall Ben Lipkin Roman Soletskyi Shengyi Costa Huang Kashif Rasul
+Longhui Yu Albert Jiang Ziju Shen Zihan Qin Bin Dong Li Zhou Yann Fleureau
+Guillaume Lample Jia LI, Edward Beeching and Stanislas Polu.
+Numinamath cot, 2024b.
+URL
+https://huggingface.co/datasets/AI-MO/NuminaMath-CoT
+.
+Kang et al. [2024]
+Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, and Boxing Chen.
+Mindstar: Enhancing math reasoning in pre-trained llms at inference
+time.
+arXiv preprint arXiv:2405.16265
+, 2024.
+Kocsis and Szepesvári [2006]
+Levente Kocsis and Csaba Szepesvári.
+Bandit based monte-carlo planning.
+volume 2006, pages 282–293, 09 2006.
+ISBN 978-3-540-45375-8.
+doi:
+10.1007/11871842_29
+.
+Kumar et al. [2024]
+Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi
+Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al.
+Training language models to self-correct via reinforcement learning.
+arXiv preprint arXiv:2409.12917
+, 2024.
+Lanham et al. [2023]
+Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison,
+Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion,
+et al.
+Measuring faithfulness in chain-of-thought reasoning.
+arXiv preprint arXiv:2307.13702
+, 2023.
+Li et al. [2024]
+Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng
+Zhang, and Houwen Peng.
+Common 7b language models already possess strong math capabilities.
+arXiv preprint arXiv:2403.04706
+, 2024.
+Liao et al. [2024]
+Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan.
+Mario: Math reasoning with code interpreter output–a reproducible
+pipeline.
+arXiv preprint arXiv:2401.08190
+, 2024.
+Lightman et al. [2023]
+Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy
+Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
+Let’s verify step by step.
+arXiv preprint arXiv:2305.20050
+, 2023.
+Lightman et al. [2024]
+Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker,
+Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
+Let’s verify step by step.
+In
+The Twelfth International Conference on Learning
+Representations
+, 2024.
+URL
+https://openreview.net/forum?id=v8L0pN6EOi
+.
+Liu et al. [2024]
+Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang
+Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
+Deepseek-v3 technical report.
+arXiv preprint arXiv:2412.19437
+, 2024.
+Luo et al. [2023]
+Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo
+Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang.
+Wizardmath: Empowering mathematical reasoning for large language
+models via reinforced evol-instruct.
+arXiv preprint arXiv:2308.09583
+, 2023.
+Luo et al. [2024]
+Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan
+Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al.
+Improve mathematical reasoning in language models by automated
+process supervision.
+arXiv preprint arXiv:2406.06592
+, 2024.
+Microsoft [2024]
+Microsoft.
+Phi-3-mini-4k-instruct, 2024.
+URL
+https://huggingface.co/microsoft/Phi-3-mini-4k-instruct
+.
+Noam Brown and Lightman [2024]
+Ilge Akkaya Noam Brown and Hunter Lightman.
+Openai’s noam brown, ilge akkaya and hunter lightman on o1 and
+teaching llms to reason better, 2024.
+URL
+https://www.youtube.com/watch?v=jPluSXJpdrA
+.
+OpenAI [2023]
+OpenAI.
+Gpt-4 technical report.
+2023.
+OpenAI [2024]
+OpenAI.
+Openai o1 system card.
+preprint
+, 2024.
+Ouyang et al. [2022]
+Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela
+Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
+Training language models to follow instructions with human feedback.
+Advances in Neural Information Processing Systems
+,
+35:27730–27744, 2022.
+Qi et al. [2024]
+Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang.
+Mutual reasoning makes smaller llms stronger problem-solvers.
+arXiv preprint arXiv:2408.06195
+, 2024.
+Qwen [2024a]
+Qwen.
+Qwen2-math-7b, 2024a.
+URL
+https://huggingface.co/Qwen/Qwen2-Math-7B
+.
+Qwen [2024b]
+Qwen.
+Qwen2.5-math-1.5b, 2024b.
+URL
+https://huggingface.co/Qwen/Qwen2.5-Math-1.5B
+.
+Qwen [2024c]
+Qwen.
+Qwen2.5-math-7b, 2024c.
+URL
+https://huggingface.co/Qwen/Qwen2.5-Math-7B
+.
+Renze and Guven [2024]
+Matthew Renze and Erhan Guven.
+Self-reflection in llm agents: Effects on problem-solving
+performance.
+arXiv preprint arXiv:2405.06682
+, 2024.
+Shinn et al. [2024]
+Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu
+Yao.
+Reflexion: Language agents with verbal reinforcement learning.
+Advances in Neural Information Processing Systems
+, 36, 2024.
+Silver et al. [2017]
+David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew
+Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore
+Graepel, et al.
+Mastering chess and shogi by self-play with a general reinforcement
+learning algorithm.
+arXiv preprint arXiv:1712.01815
+, 2017.
+Snell et al. [2024]
+Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
+Scaling llm test-time compute optimally can be more effective than
+scaling model parameters.
+arXiv preprint arXiv:2408.03314
+, 2024.
+Tang et al. [2024]
+Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei.
+Mathscale: Scaling instruction tuning for mathematical reasoning.
+arXiv preprint arXiv:2403.02884
+, 2024.
+Team [2024a]
+Qwen Team.
+Qwq: Reflect deeply on the boundaries of the unknown, November
+2024a.
+URL
+https://qwenlm.github.io/blog/qwq-32b-preview/
+.
+Team [2024b]
+The Mistral AI Team.
+Mathstral-7b-v0.1, 2024b.
+URL
+https://huggingface.co/mistralai/Mathstral-7B-v0.1
+.
+Toshniwal et al. [2024]
+Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan
+Ayrapetyan, and Igor Gitman.
+Openmathinstruct-2: Accelerating ai for math with massive open-source
+instruction data.
+arXiv preprint arXiv:2410.01560
+, 2024.
+Valmeekam et al. [2023]
+Karthik Valmeekam, Sarath Sreedharan, Matthew Marquez, Alberto Olmo, and
+Subbarao Kambhampati.
+On the planning abilities of large language models (a critical
+investigation with a proposed benchmark).
+arXiv preprint arXiv:2302.06706
+, 2023.
+Wang et al. [2024a]
+Chaojie Wang, Yanchen Deng, Zhiyi Lv, Shuicheng Yan, and An Bo.
+Q*: Improving multi-step reasoning for llms with deliberative
+planning, 2024a.
+Wang et al. [2024b]
+Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui
+Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li.
+Mathcoder: Seamless code integration in LLMs for enhanced
+mathematical reasoning.
+In
+The Twelfth International Conference on Learning
+Representations
+, 2024b.
+URL
+https://openreview.net/forum?id=z8TW0ttBPp
+.
+Wang et al. [2024c]
+Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen,
+Y. Wu, and Zhifang Sui.
+Math-shepherd: Verify and reinforce llms step-by-step without human
+annotations, 2024c.
+Wang et al. [2023]
+Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang,
+Aakanksha Chowdhery, and Denny Zhou.
+Self-consistency improves chain of thought reasoning in language
+models.
+In
+The Eleventh International Conference on Learning
+Representations
+, 2023.
+URL
+https://openreview.net/forum?id=1PL1NIMMrw
+.
+Weisstein [a]
+Eric W. Weisstein.
+Fermat’s little theorem, a.
+URL
+https://mathworld.wolfram.com/FermatsLittleTheorem.html
+.
+Weisstein [b]
+Eric W. Weisstein.
+Vieta’s formulas, from mathworld—a wolfram web resource,
+b.
+URL
+http://mathworld.wolfram.com/Tree.html
+.
+Wu et al. [2024]
+Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang.
+An empirical analysis of compute-optimal inference for
+problem-solving with language models.
+arXiv preprint arXiv:2408.00724
+, 2024.
+Xin et al. [2024]
+Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong
+Ruan, Wenda Li, and Xiaodan Liang.
+Deepseek-prover: Advancing theorem proving in llms through
+large-scale synthetic data.
+arXiv preprint arXiv:2405.14333
+, 2024.
+Yang et al. [2024]
+An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li,
+Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al.
+Qwen2. 5-math technical report: Toward mathematical expert model via
+self-improvement.
+arXiv preprint arXiv:2409.12122
+, 2024.
+Yao et al. [2024]
+Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and
+Karthik Narasimhan.
+Tree of thoughts: Deliberate problem solving with large language
+models.
+Advances in Neural Information Processing Systems
+, 36, 2024.
+Yu et al. [2023a]
+Fei Yu, Anningzhe Gao, and Benyou Wang.
+Outcome-supervised verifiers for planning in mathematical reasoning.
+arXiv preprint arXiv:2311.09724
+, 2023a.
+Yu et al. [2023b]
+Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang,
+James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.
+Metamath: Bootstrap your own mathematical questions for large
+language models.
+arXiv preprint arXiv:2309.12284
+, 2023b.
+Yuan et al. [2023]
+Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan,
+Chang Zhou, and Jingren Zhou.
+Scaling relationship on learning mathematical reasoning with large
+language models.
+arXiv preprint arXiv:2308.01825
+, 2023.
+Zhang et al. [2024a]
+Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang.
+Rest-mcts*: Llm self-training via process reward guided tree search.
+arXiv preprint arXiv:2406.03816
+, 2024a.
+Zhang et al. [2024b]
+Di Zhang, Jiatong Li, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli
+Ouyang.
+Accessing gpt-4 level mathematical olympiad solutions via monte carlo
+tree self-refine with llama-3 8b.
+arXiv preprint arXiv:2406.07394
+, 2024b.
+Zheng et al. [2023]
+Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao
+Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E.
+Gonzalez, and Ion Stoica.
+Judging LLM-as-a-judge with MT-bench and chatbot arena.
+In
+Thirty-seventh Conference on Neural Information Processing
+Systems Datasets and Benchmarks Track
+, 2023.
+Appendix A
+Appendix
+A.1
+Additional Experiments and Details
+Data Generation Details
+. As detailed in Sec.
+3.4
+, each round starts by self-generating step-by-step verified trajectories for 747k math word problems. The maximum tree depth
+d
+d
+is set to 16, with 16 MCTS rollouts conducted per problem by default. At each step, we allow to explore 8 candidate nodes, and the constant
+c
+c
+in Eq.
+1
+is set to 2 to promote greater exploration. In the bootstrap round, due to the large size of the initial policy model (236B), we used smaller parameters: 8 rollouts and 5 candidate nodes per step. To improve the accuracy of solving challenging problems in round 4, we increase the number of candidate nodes to 16 and conduct 2 MCTS tree expansions per problem using different random seeds. Detailed prompts are available in Appendix
+A.3
+.
+Training Details
+. In each round, we collect step-by-step verified trajectories to fine-tune the policy LLM and train the PPM. To reduce noise
+in synthetic math problems (e.g., incorrect ground-truth answers labeled by GPT-4), we remove synthetic problems with trajectories achieving less than 50% accuracy. Based on our extensive experiments, the policy LLM is fine-tuned from the initial base model in each round, rather than training incrementally on the model from the previous round.
+All policy SLMs are trained for 2 epochs with a sequence length of 4096 tokens and a batch size of 128. We use AdamW optimizer with a linear learning rate scheduler, setting the initial learning rate to 7e-6 for Qwen models, and a cosine scheduler with an initial learning rate of 5e-6 for Phi3-mini-Instruct.
+The PPM is trained for 1 epoch with a batch size of 512 and an initial learning rate of 7e-6.
+Training the ORM and PQM
+. The Outcome Reward Model (ORM) and the Q-value-based Process Reward Model (PQM) share the same model architecture and training parameters with our PPM. To train the ORM, we collect trajectories from math problems containing both correct and incorrect solutions. Specifically, the two trajectories with the highest average Q-values are selected as positive examples, while the two with the lowest are chosen as negative examples. Following Qwen2.5-Math
+(Yang et al.,
+2024
+)
+, we adopt the pairwise ranking loss
+(Ouyang et al.,
+2022
+)
+to optimize the ORM. To train the PQM, we follow
+Chen et al. (
+2024
+)
+to use step-level Q-values as reward labels. Let
+𝐱
+=
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+d
+\mathbf{x}=x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{d}
+be the trajectory, with annotated Q-values
+𝐐
+=
+(
+Q
+​
+(
+s
+1
+)
+,
+Q
+​
+(
+s
+1
+)
+,
+…
+,
+Q
+​
+(
+s
+d
+)
+)
+\mathbf{Q}=(Q(s_{1}),Q(s_{1}),...,Q(s_{d}))
+and predicted Q-values
+𝐐
+′
+=
+(
+Q
+′
+​
+(
+s
+1
+)
+,
+Q
+′
+​
+(
+s
+1
+)
+,
+…
+,
+Q
+′
+​
+(
+s
+d
+)
+)
+\mathbf{Q^{\prime}}=(Q^{\prime}(s_{1}),Q^{\prime}(s_{1}),...,Q^{\prime}(s_{d}))
+for each step. To stabilize PQM training, we treat each trajectory as a single training sample and predict Q-values for all steps simultaneously, rather than splitting it into individual per-step samples. Specifically, to predict the Q-value
+Q
+′
+​
+(
+s
+i
+)
+Q^{\prime}(s_{i})
+for step
+s
+i
+s_{i}
+, PQM takes the trajectory from the question up to step
+s
+i
+s_{i}
+(i.e.,
+x
+⊕
+s
+1
+⊕
+s
+2
+⊕
+…
+⊕
+s
+i
+x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{i}
+) as input and outputs a value between -1 and 1. We use a mean squared error (MSE) loss for PQM training:
+ℒ
+p
+​
+r
+​
+m
+​
+(
+𝐱
+)
+=
+‖
+𝐐
+−
+𝐐
+′
+‖
+𝟐
+\mathcal{L}_{prm}(\bf{x})=\|\bf{Q}-\bf{Q^{\prime}}\|^{2}
+(6)
+Self-evolution Inference Costs.
+In the initial bootstrap round, we use DeepSeek-Coder-v2-Instruct (236B) as the policy model, using 10 nodes of 8×80GB H100 GPUs with 8 MCTS rollouts. This required approximately two weeks to finish the data generation. For rounds 2–4, using our fine-tuned 7B SLM as the policy model, data generation was performed on 15 nodes of 4×40GB A100 GPUs,
+with each round completed in three days. In the final round, to include more challenging problems, we increased the number of MCTS rollouts to 64, extending the data generation time to one week.
+Table 9:
+Inference costs of
+\sysname
+. We show the average number of generated tokens required to generate a trajectory for a given question.
+MATH
+AIME 2024
+AMC 2023
+Olympiad Bench
+College Math
+GSM8K
+GaokaoEn 2023
+5453
+15693
+14544
+7889
+4503
+3299
+6375
+Inference Setting
+. In our evaluation, we run multiple MCTS to generate candidate solution trajectories. For each problem, we generate 32 candidate nodes at each step and use the PPM to score each node. Since the PPM effectively provides step-level quality evaluations, we limit MCTS to just 4 rollouts per step to update the Q-values. After completing MCTS, the trajectory with the highest PPM score is selected as the final answer. Table
+9
+presents the average number of tokens generated to produce a trajectory in MCTS.
+Table 10:
+Pass@1 (greedy) accuracy of our fine-tuned policy models for Phi3-mini, Qwen2.5-Math-1.5B, Qwen2-Math-7B and Qwen2.5-Math-7B.
+Model
+MATH
+AIME 2024
+AMC 2023
+Olympiad Bench
+College Math
+GSM8K
+GaokaoEn 2023
+General Base Model: Phi3-mini-Instruct (3.8B)
+Phi3-mini-Instruct
+41.4
+3.33
+7.5
+12.3
+33.1
+85.7
+37.1
+Our policy model
+68.0
+10.0
+37.5
+36.6
+48.7
+87.9
+53.2
+Math-Specialized Base Model: Qwen2.5-Math-1.5B
+Qwen2.5-Math-1.5B
+51.2
+0.0
+22.5
+16.7
+38.4
+74.6
+46.5
+Qwen2.5-Math-1.5B-Instruct
+60.0
+10.0
+60.0
+38.1
+47.7
+84.8
+65.5
+Our policy model
+74.8
+13.3
+47.5
+42.5
+50.1
+83.1
+58.7
+Math-Specialized Base Model: Qwen2-Math-7B
+Qwen2-Math-7B
+53.4
+3.3
+25.0
+17.3
+39.4
+80.4
+47.3
+Qwen2-Math-7B-Instruct
+73.2
+13.3
+62.5
+38.2
+45.9
+89.9
+62.1
+Our policy model
+73.8
+16.7
+45.0
+43.9
+52.0
+88.3
+65.2
+Math-Specialized Base Model: Qwen2.5-Math-7B
+Qwen2.5-Math-7B
+58.8
+0.0
+22.5
+21.8
+41.6
+91.6
+51.7
+Qwen2.5-Math-7B-Instruct
+82.6
+6.0
+62.5
+41.6
+46.8
+95.2
+66.8
+Our policy model
+78.4
+26.7
+47.5
+47.1
+52.5
+89.7
+65.7
+Figure 6:
+Pass@N accuracy with random sampling from different policy models. Compared to the official Qwen instruct version, our policy model exhibits a stronger ability to sample correct solutions.
+Figure 7:
+Pass@N accuracy with PPM-augmented MCTS. Under the same PPM guidance, the four policy models of varying sizes demonstrate convergent capabilities in sampling correct solutions.
+Pass@N.
+Table
+10
+compares the math reasoning performance of our policy models with the instruct versions developed by the original model team. Our policy models do not consistently outperform the instruct versions. For example, on the Qwen2.5-Math-7B base model, Qwen2.5-Math-7B-Instruct achieves 4.2% higher accuracy on the MATH benchmark. However, in System 2 deep thinking paradigm, the pass@1 accuracy alone does not fully reflect the reasoning capabilities for the policy model. To provide a more comprehensive evaluation, Fig.
+6
+and Fig.
+7
+present the pass@N accuracy. In this metric, the policy model generates
+N
+N
+solutions under two settings: random sampling (Fig.
+6
+) and PPM-augmented MCTS deep thinking (Fig.
+7
+). A problem is considered solved if one solution is correct.
+As shown in Figure
+6
+, our policy model achieves comparable Pass@64 accuracy with Qwen2.5-Math-7B-Instruct on MATH and significantly outperforms it on other benchmarks.
+Figure
+7
+further compares the pass@N accuracy of our four policy models (different sizes) after MCTS deep thinking under the same PPM guidance. We can observe that after generating 64 trajectories, the pass@N accuracy of different policy models becomes comparable.
+A.2
+Examples of Reasoning Trajectories
+Self-correction Example
+Question: The coordinates of a parallelogram are (5, 3), (6, 8), (7, 4) and
+(
+x
+,
+y
+)
+(x,y)
+and
+x
+>
+7
+x>7
+. What is the value of
+x
++
+y
+x+y
+Our trajectory
+:
+<code>
+# Import necessary libraries
+⬇
+import
+numpy
+as
+np
+<end_of_step>
+# Define the coordinates of the parallelogram
+⬇
+A
+=
+np
+.
+array
+([5,
+3])
+B
+=
+np
+.
+array
+([6,
+8])
+C
+=
+np
+.
+array
+([7,
+4])
+<end_of_step>
+# Calculate the vector differences to find the missing coordinate D
+# In a parallelogram, the vector difference between two adjacent sides is equal to the vector difference between the other two adjacent sides
+# We can use the formula: D = A + (C - B)
+⬇
+D
+=
+A
++
+(
+C
+-
+B
+)
+<end_of_step>
+# Since the coordinates provided are (5, 3), (6, 8), (7, 4) and (x, y), we can infer that the vector from B to C should be the same as the vector from A to the missing point (x, y)
+# This means: (x - 5, y - 3) = (1, -4) because the vector from B to C is (7 - 6, 4 - 8) = (1, -4)
+# Solve for x and y
+⬇
+x
+=
+5
++
+1
+y
+=
+3
++
+(-4)
+<end_of_step>
+# Verify that the point (x, y) exists and satisfies the condition x > 7
+# Since x = 6 is not greater than 7, we need to use the correct formula that maintains the parallelogram property and x > 7
+# Correcting the formula to find the other possible D [
+Self-correction!
+]
+⬇
+D
+=
+C
++
+(
+B
+-
+A
+)
+<end_of_step>
+# Solve for x and y again
+⬇
+x
+=
+7
++
+(6
+-
+5)
+y
+=
+4
++
+(8
+-
+3)
+<end_of_step>
+# Verify that the point (x, y) exists and satisfies the condition x > 7
+# Since x = 8 is greater than 7, which satisfies the condition given, we can now find the sum of x and y
+⬇
+x_plus_y
+=
+x
++
+y
+<end_of_step>
+# Now print the final answer
+⬇
+print
+(
+x_plus_y
+)
+# This should print the value of x + y
+<end_of_code>
+<output>17<end_of_output>
+<answer>The value of x + y is
+17
+<end_of_answer>
+Math statement proof using Fermat’s little theorem
+Question: If ( pt+a ) is a perfect square for some value of ( t ), then ( a
+p
+−
+1
+2
+{}^{\frac{p-1}{2}}
+- 1 ) is divisible by ( p ) (where ( p ) is a prime number).
+Our trajectory for proving
+:
+<code>
+# Step 1: Define the variables
+⬇
+from
+sympy
+import
+symbols
+,
+isprime
+,
+sqrt
+,
+mod_inverse
+# Define the variables
+⬇
+p
+,
+t
+,
+a
+=
+symbols
+(’
+p
+t
+a
+’)\\
+<end_of_step>
+# Step 2: Assume p*t + a is a perfect square
+# This means there exists an integer k such that p*t + a = k
+2
+# Rearrange to find a in terms of p, t, and k
+# a = k
+2
+- p*t
+<end_of_step>
+#
+Step 3: Use Fermat’s Little Theorem
+# Fermat’s Little Theorem states that if p is a prime and a is an integer not divisible by p,
+then a
+(p-1)
+≡
+\equiv
+1 (mod p)
+# We need to show that a
+((p-1)/2)
+- 1 is divisible by p
+<end_of_step>
+# Step 4: Substitute a = k
+2
+- p*t into Fermat’s Little Theorem
+# a
+((p-1)/2)
+≡
+\equiv
+(k
+2
+- p*t)
+((p-1)/2)
+(mod p)
+# Since p*t is divisible by p, (k
+2
+- p*t)
+((p-1)/2)
+≡
+\equiv
+k
+(p-1)
+(mod p)
+# By Fermat’s Little Theorem, k
+(p-1)
+≡
+\equiv
+1 (mod p)
+<end_of_step>
+# Step 5: Therefore, a
+((p-1)/2)
+≡
+\equiv
+1 (mod p)
+# This implies a
+((p-1)/2)
+- 1
+≡
+\equiv
+0 (mod p)
+# Hence, a
+((p-1)/2)
+- 1 is divisible by p
+<end_of_step>
+# Now print the final answer
+print("The expression a
+((p-1)/2)
+- 1 is divisible by p.")
+<end_of_code>
+<output>
+The expression
+a
+(
+(
+p
+−
+1
+)
+/
+2
+)
+a^{((p-1)/2)}
+- 1 is divisible by p.
+The statement is true.
+<end_of_output>
+Problem-solving with application of the Vieta’s formulas and the AM-GM inequality
+Question: Let
+a
+a
+be a positive real number such that all the roots of
+x
+3
++
+a
+​
+x
+2
++
+a
+​
+x
++
+1
+=
+0
+x^{3}+ax^{2}+ax+1=0
+are real. Find the smallest possible value of
+a
+.
+a.
+Our trajectory for proving
+:
+<code>
+# Step 1: Define the polynomial equation
+from sympy import symbols, solve, Eq
+# Define the variable and the parameter
+x, a = symbols(’x a’)
+<end_of_step>
+[PPM Score: -0.46430]
+# Step 2: Define the polynomial equation
+polynomial = x**3 + a*x**2 + a*x + 1
+<end_of_step>
+[PPM Score: 0.53937]
+#
+Step 3: Use Vieta’s formulas to relate the coefficients to the roots
+# Since the polynomial is monic (leading coefficient is 1), the sum of the roots (taken one at a time) is -a
+# The sum of the products of the roots taken two at a time is a
+# The product of the roots is -1
+# Since all roots are real, we can denote them as r1, r2, r3
+# From Vieta’s formulas:
+# r1 + r2 + r3 = -a
+# r1*r2 + r2*r3 + r3*r1 = a
+# r1*r2*r3 = -1
+<end_of_step>
+[PPM Score:
+0.9989]
+#
+Step 4: Use the AM-GM inequality to find the minimum value of a
+# By AM-GM, (r1*r2 + r2*r3 + r3*r1) / 3 >= (r1*r2*r3)
+(2/3)
+# Substituting the known values:
+# a / 3 >= (-1)
+(2/3)
+# a / 3 >= 1
+# a >= 3
+<end_of_step>
+[PPM Score:
+0.9999]
+# Step 5: Verify if a = 3 is achievable
+# If a = 3, then r1 + r2 + r3 = -3 and r1*r2 + r2*r3 + r3*r1 = 3
+# We need to check if there exist real roots that satisfy these conditions
+# Consider the case where r1 = r2 = r3 = -1
+# Then r1 + r2 + r3 = -3 and r1*r2 + r2*r3 + r3*r1 = 3, and r1*r2*r3 = -1
+# Thus, a = 3 is achievable with real roots
+<end_of_step>
+[PPM Score: 1.0]
+# Step 6: Print the final answer
+print(3)
+<end_of_code>
+<output>3<end_of_output>
+<answer>The smallest possible value of a is
+3
+.
+<end_of_answer>
+Problem-solving with application the Shoelace Theorem
+Question: In the diagram,
+D
+D
+and
+E
+E
+are the midpoints of
+A
+​
+B
+¯
+\overline{AB}
+and
+B
+​
+C
+¯
+\overline{BC}
+respectively. Determine the area of quadrilateral
+D
+​
+B
+​
+E
+​
+F
+DBEF
+. [asy]
+size(180); defaultpen(linewidth(.7pt)+fontsize(10pt));
+pair A, B, C, D, E, F;
+A=(0,6);
+B=(0,0);
+C=(8,0);
+D=(0,3);
+E=(4,0);
+F=(8/3,2);
+draw(E–A–C–D);
+draw((-1,0)–(10,0), EndArrow);
+draw((0,-1)–(0,8), EndArrow);
+label("
+A
+​
+(
+0
+,
+6
+)
+A(0,6)
+", A, W);
+label("
+B
+​
+(
+0
+,
+0
+)
+B(0,0)
+", B, SW);
+label("
+C
+​
+(
+8
+,
+0
+)
+C(8,0)
+", C, S);
+label("
+D
+D
+", D, W);
+label("
+E
+E
+", E, S);
+label("
+F
+F
+", F, SW);
+label("
+x
+x
+", (10,0), dir(0));
+label("
+y
+y
+", (0,8), dir(90));
+[/asy]
+Our trajectory for proving
+:
+<code>
+# Step 1: Define the coordinates of points A, B, C, D, and E
+⬇
+A
+=
+(0,
+6)
+B
+=
+(0,
+0)
+C
+=
+(8,
+0)
+D
+=
+(0,
+3)
+#
+Midpoint
+of
+AB
+E
+=
+(4,
+0)
+#
+Midpoint
+of
+BC
+<end_of_step>
+[PPM Score: -0.76639]
+# Step 2: Define the coordinates of point F using the section formula (known to be (8/3, 2))
+⬇
+F
+=
+(8/3,
+2)
+<end_of_step>
+[PPM Score: 0.8429]
+#
+Step 3: Use the Shoelace Theorem to find the area of quadrilateral DBEF
+# Shoelace Theorem formula: Area = 0.5 * |x1y2 + x2y3 + x3y4 + x4y1 - (y1x2 + y2x3 + y3x4 + y4x1)|
+# Coordinates of D, B, E, F are (0, 3), (0, 0), (4, 0), (8/3, 2) respectively
+⬇
+x1
+,
+y1
+=
+D
+x2
+,
+y2
+=
+B
+x3
+,
+y3
+=
+E
+x4
+,
+y4
+=
+F
+area
+=
+0.5
+*
+abs
+(
+x1
+*
+y2
++
+x2
+*
+y3
++
+x3
+*
+y4
++
+x4
+*
+y1
+-
+(
+y1
+*
+x2
++
+y2
+*
+x3
++
+y3
+*
+x4
++
+y4
+*
+x1
+))
+<end_of_step>
+[PPM Score:
+0.9972]
+# Step 4: Print the area of quadrilateral DBEF
+⬇
+print
+(
+area
+)
+<end_of_code>
+<output>8.0<end_of_output>
+<answer>
+8
+<end_of_answer>
+A.3
+Prompt Examples
+MCTS Prompt for DeepSeek-V2-Coder-Instruct (bootstrap round)
+You are a powerful agent with broad math knowledge and great python programming skills. You need to use python interpreter to do accurate calculation on math equations.
+!!! Remember:
+1. Use code solve the problem step by step. The solution should include three parts: <code>, <output>, and <answer>.
+2. All calculations should be done in python code. Provide concise reasoning and thinking in the comments of the code.
+3. The most related python packages include ‘math‘, ‘sympy‘, ‘scipy‘, and ‘numpy‘.
+4. Please use the following template:
+Question: the input question
+<code>Construct the code step by step. Use <end_of_step> to indicate the end of each step. Ensure your code can execute correctly(excluding <end_of_step>) and print the answer. Avoid undefined variables (NameError), unimported packages, or formatting errors (SyntaxError, TypeError). In the last step of the code, print the final answer and add a comment: Now print the final answer.<end_of_code>
+<output>Execute the code in using the Python interpreter and display the printed results.<end_of_output>
+<answer>The concise answer without verbose context, put your final answer’s numerical part (without unit, only focus on the numerical part if it’s a choice question) in
+boxed.<end_of_answer> Now! It’s your turn.
+Question:
+{input}
+The following are 2 demonstration examples:
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+<code>
+# Step 1: Calculate the total weight lifted with two 20-pound weights
+total_weight_20 = 2 * 20 * 12
+<end_of_step>
+# Step 2: Calculate the weight lifted per repetition with two 15-pound weights
+weight_per_rep_15 = 2 * 15
+<end_of_step>
+# Step 3: Calculate the number of repetitions needed to lift the same total weight with two 15-pound weights
+reps_needed = total_weight_20 / weight_per_rep_15
+<end_of_step>
+# Now print the final answer
+print(reps_needed)
+<end_of_code>
+<output>16.0 <end_of_output> <answer>From the result, we can see that Terrell must lift the 15-pound weights
+boxed16 times to lift the same total weight.
+<end_of_answer>,
+Question: Find the value of
+x
+x
+that satisfies
+3
+​
+x
++
+5
+6
+​
+x
++
+5
+=
+5
+3
+\frac{\sqrt{3x+5}}{\sqrt{6x+5}}=\frac{\sqrt{5}}{3}
+. Express your answer as a common fraction.
+<code>
+from sympy import symbols, Eq, solve, sqrt
+# Define the variable x
+x = symbols(’x’)
+<end_of_step>
+# Define the equation
+equation = Eq(sqrt(3*x + 5) / sqrt(6*x + 5), sqrt(5) / 3)
+<end_of_step>
+# Solve the equation for x
+solution = solve(equation, x)
+<end_of_step>
+# Now print the final answer
+print(solution)
+<end_of_code>
+<output>[20/3]
+<end_of_output> <answer>From the result, we can see that the value of x is
+20
+3
+<end_of_answer>
+MCTS Prompt for our policy SLM
+MCTS prompt for next step generation
+:
+<|user|>:
+{input question}
+<|assistant|>: Let’s think step by step and solve the problem with code.
+{trajectory of previous steps}
+SFT Prompt Example
+:
+<|user|>:
+How many numbers can you get by multiplying two or more distinct members of the set
+1
+,
+2
+,
+3
+,
+5
+,
+11
+\\
+{1,2,3,5,11\\
+}
+together?
+<|assistant|>: Let’s think step by step and solve the problem with code.
+<code>
+# Step 1: Create the set of numbers
+⬇
+numbers
+=
+{1,
+2,
+3,
+5,
+11}
+<end_of_step>
+# Step 2: Define a function to generate all possible products of two or more distinct members of the set
+⬇
+from
+itertools
+import
+combinations
+def
+generate_products
+(
+numbers
+):
+products
+=
+set
+()
+for
+r
+in
+range
+(2,
+len
+(
+numbers
+)
++
+1):
+#
+For
+combinations
+of
+2,
+3,
+4,
+and
+5
+elements
+for
+combo
+in
+combinations
+(
+numbers
+,
+r
+):
+product
+=
+1
+for
+num
+in
+combo
+:
+product
+*=
+num
+products
+.
+add
+(
+product
+)
+return
+products
+<end_of_step>
+# Step 3: Calculate the products
+⬇
+unique_products
+=
+generate_products
+(
+numbers
+)
+<end_of_step>
+# Step 4: Calculate the number of unique products
+⬇
+num_unique_products
+=
+len
+(
+unique_products
+)
+<end_of_step>
+# Now print the final answer
+⬇
+print
+(
+num_unique_products
+)
+<end_of_code>
+<output>15<end_of_output>
+<answer>From the result, we can see that there are
+15
+unique products.
+<end_of_answer>
+◄
+Feeling
+lucky?
+Conversion
+report
+Report
+an issue
+View original
+on arXiv
+►

research/notes/250109136-agentic-retrieval-augmented-generation-a-survey-on-agentic-rag.md

@@ -0,0 +1,203 @@
+---
+title: '[2501.09136] Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG'
+id: 250109136-agentic-retrieval-augmented-generation-a-survey-on-agentic-rag
+tags:
+- deepread
+created: '2026-06-10T00:24:47.557837Z'
+source: https://arxiv.org/abs/2501.09136
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:24:47.557707Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2501.09136] Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG
+Computer Science > Artificial Intelligence
+arXiv:2501.09136
+(cs)
+[Submitted on 15 Jan 2025 (
+v1
+), last revised 1 Apr 2026 (this version, v4)]
+Title:
+Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG
+Authors:
+Aditi Singh
+,
+Abul Ehtesham
+,
+Saket Kumar
+,
+Tala Talaei Khoei
+,
+Athanasios V. Vasilakos
+View a PDF of the paper titled Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG, by Aditi Singh and 4 other authors
+View PDF
+HTML (experimental)
+Abstract:
+Large Language Models (LLMs) have advanced artificial intelligence by enabling human-like text generation and natural language understanding. However, their reliance on static training data limits their ability to respond to dynamic, real-time queries, resulting in outdated or inaccurate outputs. Retrieval-Augmented Generation (RAG) has emerged as a solution, enhancing LLMs by integrating real-time data retrieval to provide contextually relevant and up-to-date responses. Despite its promise, traditional RAG systems are constrained by static workflows and lack the adaptability required for multi-step reasoning and complex task management. Agentic Retrieval-Augmented Generation (Agentic RAG) transcends these limitations by embedding autonomous AI agents into the RAG pipeline. These agents leverage agentic design patterns reflection, planning, tool use, and multi-agent collaboration to dynamically manage retrieval strategies, iteratively refine contextual understanding, and adapt workflows through operational structures ranging from sequential steps to adaptive collaboration. This integration enables Agentic RAG systems to deliver flexibility, scalability, and context-awareness across diverse applications. This paper presents an analytical survey of Agentic RAG systems. It traces the evolution of RAG paradigms, introduces a principled taxonomy of Agentic RAG architectures based on agent cardinality, control structure, autonomy, and knowledge representation, and provides a comparative analysis of design trade-offs across existing frameworks. The survey examines applications in healthcare, finance, education, and enterprise document processing, and distills practical lessons for system designers and practitioners. Finally, it identifies key open research challenges related to evaluation, coordination, memory management, efficiency, and governance, outlining directions for future research.
+Subjects:
+Artificial Intelligence (cs.AI)
+; Computation and Language (cs.CL); Information Retrieval (cs.IR)
+Cite as:
+arXiv:2501.09136
+[cs.AI]
+(or
+arXiv:2501.09136v4
+[cs.AI]
+for this version)
+https://doi.org/10.48550/arXiv.2501.09136
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Abul Ehtesham [
+view email
+]
+[v1]
+Wed, 15 Jan 2025 20:40:25 UTC (20,962 KB)
+[v2]
+Mon, 3 Feb 2025 04:01:36 UTC (22,453 KB)
+[v3]
+Tue, 4 Feb 2025 04:48:00 UTC (22,430 KB)
+[v4]
+Wed, 1 Apr 2026 15:51:06 UTC (13,996 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG, by Aditi Singh and 4 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.AI
+< prev
+|
+next >
+new
+|
+recent
+|
+2025-01
+Change to browse by:
+cs
+cs.CL
+cs.IR
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+Links to Code Toggle
+Papers with Code
+(
+What is Papers with Code?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/250109891-evolving-deeper-llm-thinking.md

@@ -0,0 +1,196 @@
+---
+title: '[2501.09891] Evolving Deeper LLM Thinking'
+id: 250109891-evolving-deeper-llm-thinking
+tags:
+- deepread
+created: '2026-06-10T00:24:58.674469Z'
+source: https://arxiv.org/abs/2501.09891
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:24:58.674344Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2501.09891] Evolving Deeper LLM Thinking
+Computer Science > Artificial Intelligence
+arXiv:2501.09891
+(cs)
+[Submitted on 17 Jan 2025]
+Title:
+Evolving Deeper LLM Thinking
+Authors:
+Kuang-Huei Lee
+,
+Ian Fischer
+,
+Yueh-Hua Wu
+,
+Dave Marwood
+,
+Shumeet Baluja
+,
+Dale Schuurmans
+,
+Xinyun Chen
+View a PDF of the paper titled Evolving Deeper LLM Thinking, by Kuang-Huei Lee and 6 other authors
+View PDF
+HTML (experimental)
+Abstract:
+We explore an evolutionary search strategy for scaling inference time compute in Large Language Models. The proposed approach, Mind Evolution, uses a language model to generate, recombine and refine candidate responses. The proposed approach avoids the need to formalize the underlying inference problem whenever a solution evaluator is available. Controlling for inference cost, we find that Mind Evolution significantly outperforms other inference strategies such as Best-of-N and Sequential Revision in natural language planning tasks. In the TravelPlanner and Natural Plan benchmarks, Mind Evolution solves more than 98% of the problem instances using Gemini 1.5 Pro without the use of a formal solver.
+Subjects:
+Artificial Intelligence (cs.AI)
+Cite as:
+arXiv:2501.09891
+[cs.AI]
+(or
+arXiv:2501.09891v1
+[cs.AI]
+for this version)
+https://doi.org/10.48550/arXiv.2501.09891
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Dale Schuurmans [
+view email
+]
+[v1]
+Fri, 17 Jan 2025 00:41:44 UTC (3,183 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Evolving Deeper LLM Thinking, by Kuang-Huei Lee and 6 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.AI
+< prev
+|
+next >
+new
+|
+recent
+|
+2025-01
+Change to browse by:
+cs
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+Links to Code Toggle
+Papers with Code
+(
+What is Papers with Code?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/250112599-kimi-k15-scaling-reinforcement-learning-with-llms.md

@@ -0,0 +1,386 @@
+---
+title: '[2501.12599] Kimi k1.5: Scaling Reinforcement Learning with LLMs'
+id: 250112599-kimi-k15-scaling-reinforcement-learning-with-llms
+tags:
+- deepread
+created: '2026-06-10T00:24:53.655467Z'
+source: https://arxiv.org/abs/2501.12599
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:24:53.655188Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2501.12599] Kimi k1.5: Scaling Reinforcement Learning with LLMs
+Computer Science > Artificial Intelligence
+arXiv:2501.12599
+(cs)
+[Submitted on 22 Jan 2025 (
+v1
+), last revised 3 Jun 2025 (this version, v4)]
+Title:
+Kimi k1.5: Scaling Reinforcement Learning with LLMs
+Authors:
+Kimi Team
+,
+Angang Du
+,
+Bofei Gao
+,
+Bowei Xing
+,
+Changjiu Jiang
+,
+Cheng Chen
+,
+Cheng Li
+,
+Chenjun Xiao
+,
+Chenzhuang Du
+,
+Chonghua Liao
+,
+Chuning Tang
+,
+Congcong Wang
+,
+Dehao Zhang
+,
+Enming Yuan
+,
+Enzhe Lu
+,
+Fengxiang Tang
+,
+Flood Sung
+,
+Guangda Wei
+,
+Guokun Lai
+,
+Haiqing Guo
+,
+Han Zhu
+,
+Hao Ding
+,
+Hao Hu
+,
+Hao Yang
+,
+Hao Zhang
+,
+Haotian Yao
+,
+Haotian Zhao
+,
+Haoyu Lu
+,
+Haoze Li
+,
+Haozhen Yu
+,
+Hongcheng Gao
+,
+Huabin Zheng
+,
+Huan Yuan
+,
+Jia Chen
+,
+Jianhang Guo
+,
+Jianlin Su
+,
+Jianzhou Wang
+,
+Jie Zhao
+,
+Jin Zhang
+,
+Jingyuan Liu
+,
+Junjie Yan
+,
+Junyan Wu
+,
+Lidong Shi
+,
+Ling Ye
+,
+Longhui Yu
+,
+Mengnan Dong
+,
+Neo Zhang
+,
+Ningchen Ma
+,
+Qiwei Pan
+,
+Qucheng Gong
+,
+Shaowei Liu
+,
+Shengling Ma
+,
+Shupeng Wei
+,
+Sihan Cao
+,
+Siying Huang
+,
+Tao Jiang
+,
+Weihao Gao
+,
+Weimin Xiong
+,
+Weiran He
+,
+Weixiao Huang
+,
+Weixin Xu
+,
+Wenhao Wu
+,
+Wenyang He
+,
+Xianghui Wei
+,
+Xianqing Jia
+,
+Xingzhe Wu
+,
+Xinran Xu
+,
+Xinxing Zu
+,
+Xinyu Zhou
+,
+Xuehai Pan
+,
+Y. Charles
+,
+Yang Li
+,
+Yangyang Hu
+,
+Yangyang Liu
+,
+Yanru Chen
+,
+Yejie Wang
+,
+Yibo Liu
+,
+Yidao Qin
+,
+Yifeng Liu
+,
+Ying Yang
+,
+Yiping Bao
+,
+Yulun Du
+,
+Yuxin Wu
+,
+Yuzhi Wang
+,
+Zaida Zhou
+,
+Zhaoji Wang
+,
+Zhaowei Li
+,
+Zhen Zhu
+,
+Zheng Zhang
+,
+Zhexu Wang
+,
+Zhilin Yang
+,
+Zhiqi Huang
+,
+Zihao Huang
+,
+Ziyao Xu
+,
+Zonghan Yang
+,
+Zongyu Lin
+View a PDF of the paper titled Kimi k1.5: Scaling Reinforcement Learning with LLMs, by Kimi Team and 95 other authors
+View PDF
+HTML (experimental)
+Abstract:
+Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).
+Comments:
+25 pages
+Subjects:
+Artificial Intelligence (cs.AI)
+; Machine Learning (cs.LG)
+Cite as:
+arXiv:2501.12599
+[cs.AI]
+(or
+arXiv:2501.12599v4
+[cs.AI]
+for this version)
+https://doi.org/10.48550/arXiv.2501.12599
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Flood Sung [
+view email
+]
+[v1]
+Wed, 22 Jan 2025 02:48:14 UTC (614 KB)
+[v2]
+Wed, 5 Mar 2025 02:16:32 UTC (614 KB)
+[v3]
+Wed, 28 May 2025 03:57:30 UTC (614 KB)
+[v4]
+Tue, 3 Jun 2025 02:14:54 UTC (603 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Kimi k1.5: Scaling Reinforcement Learning with LLMs, by Kimi Team and 95 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.AI
+< prev
+|
+next >
+new
+|
+recent
+|
+2025-01
+Change to browse by:
+cs
+cs.LG
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+Links to Code Toggle
+Papers with Code
+(
+What is Papers with Code?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/250118512-streaming-diloco-with-overlapping-communication-towards-a-distributed.md

@@ -0,0 +1,206 @@
+---
+title: '[2501.18512] Streaming DiLoCo with overlapping communication: Towards a Distributed
+  Free Lunch'
+id: 250118512-streaming-diloco-with-overlapping-communication-towards-a-distributed
+tags:
+- deepread
+created: '2026-06-10T00:30:21.211856Z'
+source: https://arxiv.org/abs/2501.18512
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:30:21.211709Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2501.18512] Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
+Computer Science > Computation and Language
+arXiv:2501.18512
+(cs)
+[Submitted on 30 Jan 2025]
+Title:
+Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
+Authors:
+Arthur Douillard
+,
+Yanislav Donchev
+,
+Keith Rush
+,
+Satyen Kale
+,
+Zachary Charles
+,
+Zachary Garrett
+,
+Gabriel Teston
+,
+Dave Lacey
+,
+Ross McIlroy
+,
+Jiajun Shen
+,
+Alexandre Ramé
+,
+Arthur Szlam
+,
+Marc'Aurelio Ranzato
+,
+Paul Barham
+View a PDF of the paper titled Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch, by Arthur Douillard and Yanislav Donchev and Keith Rush and Satyen Kale and Zachary Charles and Zachary Garrett and Gabriel Teston and Dave Lacey and Ross McIlroy and Jiajun Shen and Alexandre Ram\'e and Arthur Szlam and Marc'Aurelio Ranzato and Paul Barham
+View PDF
+HTML (experimental)
+Abstract:
+Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time. Since internal states and parameter gradients need to be exchanged at each and every single gradient step, all devices need to be co-located using low-latency high-bandwidth communication links to support the required high volume of exchanged bits. Recently, distributed algorithms like DiLoCo have relaxed such co-location constraint: accelerators can be grouped into ``workers'', where synchronizations between workers only occur infrequently. This in turn means that workers can afford being connected by lower bandwidth communication links without affecting learning quality. However, in these methods, communication across workers still requires the same peak bandwidth as before, as the synchronizations require all parameters to be exchanged across all workers. In this paper, we improve DiLoCo in three ways. First, we synchronize only subsets of parameters in sequence, rather than all at once, which greatly reduces peak bandwidth. Second, we allow workers to continue training while synchronizing, which decreases wall clock time. Third, we quantize the data exchanged by workers, which further reduces bandwidth across workers. By properly combining these modifications, we show experimentally that we can distribute training of billion-scale parameters and reach similar quality as before, but reducing required bandwidth by two orders of magnitude.
+Subjects:
+Computation and Language (cs.CL)
+Cite as:
+arXiv:2501.18512
+[cs.CL]
+(or
+arXiv:2501.18512v1
+[cs.CL]
+for this version)
+https://doi.org/10.48550/arXiv.2501.18512
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Arthur Douillard [
+view email
+]
+[v1]
+Thu, 30 Jan 2025 17:23:50 UTC (3,278 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch, by Arthur Douillard and Yanislav Donchev and Keith Rush and Satyen Kale and Zachary Charles and Zachary Garrett and Gabriel Teston and Dave Lacey and Ross McIlroy and Jiajun Shen and Alexandre Ram\'e and Arthur Szlam and Marc'Aurelio Ranzato and Paul Barham
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.CL
+< prev
+|
+next >
+new
+|
+recent
+|
+2025-01
+Change to browse by:
+cs
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/250118639-a-comprehensive-survey-of-the-lean-4-theorem-prover-architecture-appli.md

@@ -0,0 +1,179 @@
+---
+title: '[2501.18639] A Comprehensive Survey of the Lean 4 Theorem Prover: Architecture,
+  Applications, and Advances'
+id: 250118639-a-comprehensive-survey-of-the-lean-4-theorem-prover-architecture-appli
+tags:
+- deepread
+created: '2026-06-10T00:25:14.929249Z'
+source: https://arxiv.org/abs/2501.18639
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:25:14.929112Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2501.18639] A Comprehensive Survey of the Lean 4 Theorem Prover: Architecture, Applications, and Advances
+Computer Science > Logic in Computer Science
+arXiv:2501.18639
+(cs)
+[Submitted on 28 Jan 2025]
+Title:
+A Comprehensive Survey of the Lean 4 Theorem Prover: Architecture, Applications, and Advances
+Authors:
+Xichen Tang
+View a PDF of the paper titled A Comprehensive Survey of the Lean 4 Theorem Prover: Architecture, Applications, and Advances, by Xichen Tang
+View PDF
+Abstract:
+This comprehensive survey examines Lean 4, a state-of-the-art interactive theorem prover and functional programming language. We analyze its architectural design, type system, metaprogramming capabilities, and practical applications in formal verification and mathematics. Through detailed comparisons with other proof assistants and extensive case studies, we demonstrate Lean 4's unique advantages in proof automation, performance, and usability. The paper also explores recent developments in its ecosystem, including libraries, tools, and educational applications, providing insights into its growing impact on formal methods and mathematical formalization.
+Subjects:
+Logic in Computer Science (cs.LO)
+; Programming Languages (cs.PL)
+Cite as:
+arXiv:2501.18639
+[cs.LO]
+(or
+arXiv:2501.18639v1
+[cs.LO]
+for this version)
+https://doi.org/10.48550/arXiv.2501.18639
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Xichen Tang [
+view email
+]
+[v1]
+Tue, 28 Jan 2025 17:15:54 UTC (2,729 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled A Comprehensive Survey of the Lean 4 Theorem Prover: Architecture, Applications, and Advances, by Xichen Tang
+View PDF
+view license
+Current browse context:
+cs.LO
+< prev
+|
+next >
+new
+|
+recent
+|
+2025-01
+Change to browse by:
+cs
+cs.PL
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)

research/notes/250202047-amasquad-a-benchmark-for-amharic-extractive-question-answering.md

@@ -0,0 +1,183 @@
+---
+title: '[2502.02047] AmaSQuAD: A Benchmark for Amharic Extractive Question Answering'
+id: 250202047-amasquad-a-benchmark-for-amharic-extractive-question-answering
+tags:
+- deepread
+created: '2026-06-10T00:24:12.341628Z'
+source: https://arxiv.org/abs/2502.02047
+source_domain: arxiv.org
+fetched_at: '2026-06-10T00:24:12.341471Z'
+fetch_provider: builtin
+status: draft
+type: note
+tier: institutional
+content_type: paper
+deprecated: false
+---
+
+[2502.02047] AmaSQuAD: A Benchmark for Amharic Extractive Question Answering
+Computer Science > Computation and Language
+arXiv:2502.02047
+(cs)
+[Submitted on 4 Feb 2025]
+Title:
+AmaSQuAD: A Benchmark for Amharic Extractive Question Answering
+Authors:
+Nebiyou Daniel Hailemariam
+,
+Blessed Guda
+,
+Tsegazeab Tefferi
+View a PDF of the paper titled AmaSQuAD: A Benchmark for Amharic Extractive Question Answering, by Nebiyou Daniel Hailemariam and 2 other authors
+View PDF
+HTML (experimental)
+Abstract:
+This research presents a novel framework for translating extractive question-answering datasets into low-resource languages, as demonstrated by the creation of the AmaSQuAD dataset, a translation of SQuAD 2.0 into Amharic. The methodology addresses challenges related to misalignment between translated questions and answers, as well as the presence of multiple answer instances in the translated context. For this purpose, we used cosine similarity utilizing embeddings from a fine-tuned BERT-based model for Amharic and Longest Common Subsequence (LCS). Additionally, we fine-tune the XLM-R model on the AmaSQuAD synthetic dataset for Amharic Question-Answering. The results show an improvement in baseline performance, with the fine-tuned model achieving an increase in the F1 score from 36.55% to 44.41% and 50.01% to 57.5% on the AmaSQuAD development dataset. Moreover, the model demonstrates improvement on the human-curated AmQA dataset, increasing the F1 score from 67.80% to 68.80% and the exact match score from 52.50% to 52.66%.The AmaSQuAD dataset is publicly available Datasets
+Subjects:
+Computation and Language (cs.CL)
+Cite as:
+arXiv:2502.02047
+[cs.CL]
+(or
+arXiv:2502.02047v1
+[cs.CL]
+for this version)
+https://doi.org/10.48550/arXiv.2502.02047
+Focus to learn more
+arXiv-issued DOI via DataCite
+Submission history
+From: Blessed Guda [
+view email
+]
+[v1]
+Tue, 4 Feb 2025 06:27:39 UTC (778 KB)
+Full-text links:
+Access Paper:
+View a PDF of the paper titled AmaSQuAD: A Benchmark for Amharic Extractive Question Answering, by Nebiyou Daniel Hailemariam and 2 other authors
+View PDF
+HTML (experimental)
+TeX Source
+view license
+Current browse context:
+cs.CL
+< prev
+|
+next >
+new
+|
+recent
+|
+2025-02
+Change to browse by:
+cs
+References & Citations
+NASA ADS
+Google Scholar
+Semantic Scholar
+export BibTeX citation
+Loading...
+BibTeX formatted citation
+×
+loading...
+Data provided by:
+Bookmark
+Bibliographic Tools
+Bibliographic and Citation Tools
+Bibliographic Explorer Toggle
+Bibliographic Explorer
+(
+What is the Explorer?
+)
+Connected Papers Toggle
+Connected Papers
+(
+What is Connected Papers?
+)
+Litmaps Toggle
+Litmaps
+(
+What is Litmaps?
+)
+scite.ai Toggle
+scite Smart Citations
+(
+What are Smart Citations?
+)
+Code, Data, Media
+Code, Data and Media Associated with this Article
+alphaXiv Toggle
+alphaXiv
+(
+What is alphaXiv?
+)
+Links to Code Toggle
+CatalyzeX Code Finder for Papers
+(
+What is CatalyzeX?
+)
+DagsHub Toggle
+DagsHub
+(
+What is DagsHub?
+)
+GotitPub Toggle
+Gotit.pub
+(
+What is GotitPub?
+)
+Huggingface Toggle
+Hugging Face
+(
+What is Huggingface?
+)
+ScienceCast Toggle
+ScienceCast
+(
+What is ScienceCast?
+)
+Demos
+Demos
+Replicate Toggle
+Replicate
+(
+What is Replicate?
+)
+Spaces Toggle
+Hugging Face Spaces
+(
+What is Spaces?
+)
+Spaces Toggle
+TXYZ.AI
+(
+What is TXYZ.AI?
+)
+Related Papers
+Recommenders and Search Tools
+Link to Influence Flower
+Influence Flower
+(
+What are Influence Flowers?
+)
+Core recommender toggle
+CORE Recommender
+(
+What is CORE?
+)
+Author
+Venue
+Institution
+Topic
+About arXivLabs
+arXivLabs: experimental projects with community collaborators
+arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
+Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
+Have an idea for a project that will add value for arXiv's community?
+Learn more about arXivLabs
+.
+Which authors of this paper are endorsers?
+|
+Disable MathJax
+(
+What is MathJax?
+)