BurnyCoder/multi-agent-swe-rl-qwen3.5-2b-pilot

Local multi-agent software self-play pilot

An unchanged research adapter: the pilot completed no optimizer updates. This is not a trained checkpoint.

All primaries and asynchronous subagents use exactly one shared adapter. The frozen base weights, vision tower, embeddings and output head are not trained.

Exact base revision: 15852e8c16360a2fea060d615a32b45270f8a8fc. Selected policy version: 0. Adapter fingerprint: 9312a507ea4ffa381a7dedfb8b880b1169761ac338a899ce4ed6732b0c503958. The selected adapter has no optimizer updates; completed zero-advantage collection groups can leave it unchanged. In the saved analysis, trained identifies the final-evaluation condition, not evidence of an optimizer update.

Recorded completed collection groups: 2. Saved report status: pilot_failed. Pilot source commit: 109dfa67740ff9f0260d22be9407d047d90565f7. Report/export source commit: a4a95e81d2d753539786a2a0671db4e99050b532.

Reproduction

Use the locked research harness and its documented GPU environment for model loading. The adapter, processor, checkpoint metadata, optimizer/RNG state and experiment configuration are included. The harness loader reads the exact base revision and verifies the adapter fingerprint:

import json
from pathlib import Path
from multi_agent_ssr.contracts import ModelConfig
from multi_agent_ssr.model import LocalPolicy

bundle = Path("/path/to/downloaded-bundle")
record = json.loads((bundle / "checkpoint.json").read_text())
policy = LocalPolicy(ModelConfig(**record["model_config"]),
                     log_path=Path("reload-model.jsonl"), checkpoint=bundle)

The export pins adapter_config.json's revision for PEFT's base-model lookup. source-metadata/adapter_config.json preserves the original saved bytes, and export-transformations.json records the metadata change and before/after hashes. The source checkpoint and adapter weights are unchanged.

The harness's generation configuration unions the base model's EOS IDs with the tokenizer's chat EOS and deduplicates them; prompt encoding disables thinking. Reusing only the base generation defaults can continue past a chat turn; retain the harness configuration when sampling.

The allowlisted raw records retain exact prompt tokens, generated action tokens, behavior log probabilities, policy versions and parent identities. Stored paths are unchanged provenance metadata, not instructions to fetch excluded files. This bundle excludes repository snapshots, official scorer caches and reference fixes. It supports report regeneration and inspection of recorded actions; rerunning sampling is not guaranteed to reproduce actions bit for bit. bundle-inventory.json contains byte counts and SHA256 hashes.

From a checkout at the report/export source commit, regenerate all tables, figures, paper and dashboard without a model call (the report command also refreshes that checkout's generated README results block):

BUNDLE=/path/to/downloaded-bundle
uv run --no-sync selfplay report --run "$BUNDLE"/run-records/pilot-recovery --output "$BUNDLE/regenerated-reports" --site "$BUNDLE/regenerated-site"

Recorded evaluation

Aggregate results are shown below. Complete evaluation records contain the full saved analysis, including task-level outcomes and official reports.

{
  "conditions": {
    "human": {
      "base": {
        "recorded_tasks": 32,
        "completed": 21,
        "attempted": 22,
        "successes": 0,
        "unscored": 11,
        "rate": 0.0,
        "statuses": {
          "completed": 21,
          "timeout": 1,
          "pending": 10
        },
        "generated_tokens": 25254,
        "elapsed_seconds": 8523.812844521999,
        "delegations": 3,
        "planned": 32,
        "unrecorded": 0
      },
      "trained": {
        "recorded_tasks": 32,
        "completed": 29,
        "attempted": 30,
        "successes": 0,
        "unscored": 3,
        "rate": 0.0,
        "statuses": {
          "completed": 29,
          "interrupted_budget": 1,
          "not_run_budget": 2
        },
        "generated_tokens": 31365,
        "elapsed_seconds": 10009.669275809993,
        "delegations": 3,
        "planned": 32,
        "unrecorded": 0
      }
    },
    "synthetic": {
      "base": {
        "recorded_tasks": 16,
        "completed": 0,
        "attempted": 0,
        "successes": 0,
        "unscored": 16,
        "rate": null,
        "statuses": {
          "not_run_budget": 16
        },
        "generated_tokens": 0,
        "elapsed_seconds": 0,
        "delegations": 0,
        "planned": 16,
        "unrecorded": 0
      },
      "trained": {
        "recorded_tasks": 16,
        "completed": 0,
        "attempted": 0,
        "successes": 0,
        "unscored": 16,
        "rate": null,
        "statuses": {
          "not_run_budget": 16
        },
        "generated_tokens": 0,
        "elapsed_seconds": 0,
        "delegations": 0,
        "planned": 16,
        "unrecorded": 0
      }
    }
  },
  "comparisons": {
    "human": {
      "paired_tasks": 21,
      "task_ids": [
        "psf__requests-1142",
        "psf__requests-1766",
        "psf__requests-1921",
        "psf__requests-2931",
        "psf__requests-5414",
        "pylint-dev__pylint-4551",
        "pylint-dev__pylint-4661",
        "pylint-dev__pylint-4970",
        "pylint-dev__pylint-6386",
        "pylint-dev__pylint-6903",
        "pylint-dev__pylint-7080",
        "pytest-dev__pytest-10051",
        "pytest-dev__pytest-10356",
        "pytest-dev__pytest-5262",
        "pytest-dev__pytest-5631",
        "pytest-dev__pytest-5809",
        "pytest-dev__pytest-6202",
        "pytest-dev__pytest-7205",
        "pytest-dev__pytest-7432",
        "pytest-dev__pytest-7521",
        "pytest-dev__pytest-7982"
      ],
      "difference": 0.0,
      "ci95": [
        0.0,
        0.0
      ],
      "interval_degenerate": true,
      "improved": 0,
      "regressed": 0,
      "unchanged": 21,
      "interpretation": "no_observed_difference",
      "preregistered_decision": "contradicts_predicted_improvement"
    },
    "synthetic": {
      "paired_tasks": 0,
      "task_ids": [],
      "difference": null,
      "ci95": null,
      "interval_degenerate": false,
      "improved": 0,
      "regressed": 0,
      "unchanged": 0,
      "interpretation": "not_evaluated",
      "preregistered_decision": "not_evaluated"
    }
  },
  "bootstrap": {
    "method": "paired percentile",
    "samples": 10000,
    "seed": 0,
    "confidence": 0.95
  }
}

Human and synthetic bugs are separate evaluations. Null results are unscored, not failures or successes. The small cohort and single training seed do not establish robustness; pretraining contamination is unknown. See experiment.md and paper.pdf for coverage, paired uncertainty, limitations and negative results.

Method and licenses

This adapts Self-play SWE-RL, CWM, and OpenCode delegation. The pinned base license is Apache-2.0. Training repositories retain their MIT/BSD notices; the harness license is MIT. Benchmark repository source excerpts in raw model/tool records have separate upstream notices and pinned source associations. These include notices from requests, Flask, pylint and pytest; the cached dataset card's missing license declaration is documented in that provenance. When present, license-provenance.json identifies the exact bundled license/notice files and their source revisions. No held-out reference fix or evaluation test patch enters training.